[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:56:45 dist-test-slave-pz0h krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:56:45 dist-test-slave-pz0h krb5kdc[399](info): set up 2 sockets
Sep 27 14:56:45 dist-test-slave-pz0h krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 14:56:46.011351 392 mini_kdc.cc:121] Time spent starting KDC: real 1.733s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 14:56:46 dist-test-slave-pz0h krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:56:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:56:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:56:46.131592 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34571
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:56:46.247116 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:56:46.247422 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:56:46.247463 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:56:46.251720 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:56:46.251802 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:56:46.251821 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:56:46.251837 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:56:46.251850 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:56:46.257519 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:56:46.258710 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:56:46.259882 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:56:46.269320 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:46.269402 412 server_base.cc:1034] running on GCE node
W20260927 14:56:46.269330 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:46.269616 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:46.270246 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:56:46.271521 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:56:46.272709 412 hybrid_clock.cc:648] HybridClock initialized: now 1790521006272647 us; error 74 us; skew 500 ppm
Sep 27 14:56:46 dist-test-slave-pz0h krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:56:46.275578 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:56:46.276579 412 webserver.cc:460] Webserver started at http://127.0.98.62:40311/ using document root <none> and password file <none>
I20260927 14:56:46.277173 412 fs_manager.cc:359] Metadata directory not provided
I20260927 14:56:46.277240 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:56:46.277455 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:56:46.279803 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "5efa592c71c340c4bb09440a1016edbc"
format_stamp: "Formatted at 2026-09-27 14:56:46 on dist-test-slave-pz0h"
server_key: "ea24055b37b16534da1f9ee8e8ed053e"
server_key_iv: "79a4af5abedb2c98f1358534dbba5ffc"
server_key_version: "encryptionkey@0"
I20260927 14:56:46.280437 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "5efa592c71c340c4bb09440a1016edbc"
format_stamp: "Formatted at 2026-09-27 14:56:46 on dist-test-slave-pz0h"
server_key: "ea24055b37b16534da1f9ee8e8ed053e"
server_key_iv: "79a4af5abedb2c98f1358534dbba5ffc"
server_key_version: "encryptionkey@0"
I20260927 14:56:46.284971 412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 14:56:46.287774 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:46.289193 412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 14:56:46.289333 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "5efa592c71c340c4bb09440a1016edbc"
format_stamp: "Formatted at 2026-09-27 14:56:46 on dist-test-slave-pz0h"
server_key: "ea24055b37b16534da1f9ee8e8ed053e"
server_key_iv: "79a4af5abedb2c98f1358534dbba5ffc"
server_key_version: "encryptionkey@0"
I20260927 14:56:46.289456 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:56:46.307622 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:56:46.311002 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:56:46.311205 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:56:46.318231 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34571
I20260927 14:56:46.318274 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34571 every 8 connection(s)
I20260927 14:56:46.319334 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:56:46.322485 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:46.328593 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5efa592c71c340c4bb09440a1016edbc: Bootstrap starting.
I20260927 14:56:46.329586 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 412
I20260927 14:56:46.329767 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 14:56:46.331522 392 external_mini_cluster.cc:1428] Setting key c00e2f711d9b4f1ef035b4c2c2c72f14
I20260927 14:56:46.332113 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5efa592c71c340c4bb09440a1016edbc: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:46.333137 480 log.cc:824] T 00000000000000000000000000000000 P 5efa592c71c340c4bb09440a1016edbc: Log is configured to *not* fsync() on all Append() calls
I20260927 14:56:46.335624 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5efa592c71c340c4bb09440a1016edbc: No bootstrap required, opened a new log
I20260927 14:56:46.339697 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5efa592c71c340c4bb09440a1016edbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efa592c71c340c4bb09440a1016edbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34571 } }
I20260927 14:56:46.340013 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5efa592c71c340c4bb09440a1016edbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:46.340080 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5efa592c71c340c4bb09440a1016edbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5efa592c71c340c4bb09440a1016edbc, State: Initialized, Role: FOLLOWER
I20260927 14:56:46.340605 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5efa592c71c340c4bb09440a1016edbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efa592c71c340c4bb09440a1016edbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34571 } }
I20260927 14:56:46.340781 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5efa592c71c340c4bb09440a1016edbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:56:46.340849 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5efa592c71c340c4bb09440a1016edbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:56:46.340943 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5efa592c71c340c4bb09440a1016edbc [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:46.341914 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5efa592c71c340c4bb09440a1016edbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efa592c71c340c4bb09440a1016edbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34571 } }
I20260927 14:56:46.342298 480 leader_election.cc:302] T 00000000000000000000000000000000 P 5efa592c71c340c4bb09440a1016edbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5efa592c71c340c4bb09440a1016edbc; no voters:
I20260927 14:56:46.342625 480 leader_election.cc:288] T 00000000000000000000000000000000 P 5efa592c71c340c4bb09440a1016edbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:56:46.342720 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5efa592c71c340c4bb09440a1016edbc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:56:46.342906 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5efa592c71c340c4bb09440a1016edbc [term 1 LEADER]: Becoming Leader. State: Replica: 5efa592c71c340c4bb09440a1016edbc, State: Running, Role: LEADER
I20260927 14:56:46.343227 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5efa592c71c340c4bb09440a1016edbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efa592c71c340c4bb09440a1016edbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34571 } }
I20260927 14:56:46.344024 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5efa592c71c340c4bb09440a1016edbc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:56:46.345108 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5efa592c71c340c4bb09440a1016edbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5efa592c71c340c4bb09440a1016edbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efa592c71c340c4bb09440a1016edbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34571 } } }
I20260927 14:56:46.345119 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5efa592c71c340c4bb09440a1016edbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5efa592c71c340c4bb09440a1016edbc. Latest consensus state: current_term: 1 leader_uuid: "5efa592c71c340c4bb09440a1016edbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efa592c71c340c4bb09440a1016edbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34571 } } }
I20260927 14:56:46.345253 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5efa592c71c340c4bb09440a1016edbc [sys.catalog]: This master's current role is: LEADER
I20260927 14:56:46.345264 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5efa592c71c340c4bb09440a1016edbc [sys.catalog]: This master's current role is: LEADER
W20260927 14:56:46.349141 500 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5efa592c71c340c4bb09440a1016edbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:56:46.349246 500 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:56:46.349376 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:56:46.350148 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 14:56:46 dist-test-slave-pz0h krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:56:46.357061 494 catalog_manager.cc:1347] Generated new cluster ID: bd58a8fa9755436db2857d2aaa33554d
I20260927 14:56:46.357146 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:56:46.359957 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:46.335272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39286 (local address 127.0.98.62:34571)
0927 14:56:46.335787 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:46.335807 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:46.337058 (+ 1251us) server_negotiation.cc:407] Connection header received
0927 14:56:46.339802 (+ 2744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:46.339822 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:46.340262 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:46.340643 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:46.343170 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:46.344384 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:46.346191 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:46.346957 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:46.347111 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:46.352756 (+ 5645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:46.352762 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:46.352776 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:46.352814 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:46.355161 (+ 2347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:46.355948 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:46.355957 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:46.355965 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:46.356148 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:46.356619 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:46.356623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:46.356624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:46.357224 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:46.357446 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:46.358641 (+ 1195us) server_negotiation.cc:299] Negotiation successful
0927 14:56:46.359004 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":140,"threads_started":1}
I20260927 14:56:46.376622 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:56:46.378006 494 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:56:46.384182 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5efa592c71c340c4bb09440a1016edbc: Generated new TSK 0
I20260927 14:56:46.385039 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:56:46.426898 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34571
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:56:46.556206 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:56:46.556510 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:56:46.556556 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:56:46.560711 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:56:46.560792 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:56:46.560856 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:56:46.566712 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34571
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:56:46.567899 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:56:46.568961 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:56:46.579149 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:46.579149 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:46.579517 506 server_base.cc:1034] running on GCE node
W20260927 14:56:46.579162 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:46.580018 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:56:46.580751 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:56:46.581991 506 hybrid_clock.cc:648] HybridClock initialized: now 1790521006581969 us; error 57 us; skew 500 ppm
Sep 27 14:56:46 dist-test-slave-pz0h krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:56:46.585317 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:56:46.586382 506 webserver.cc:460] Webserver started at http://127.0.98.1:37875/ using document root <none> and password file <none>
I20260927 14:56:46.587021 506 fs_manager.cc:359] Metadata directory not provided
I20260927 14:56:46.587095 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:56:46.587301 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:56:46.589514 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-0/data/instance:
uuid: "38bb22782624453886cb350654207fc8"
format_stamp: "Formatted at 2026-09-27 14:56:46 on dist-test-slave-pz0h"
server_key: "8ffd9fb34779f9d8942ca1f94e080835"
server_key_iv: "42bb63ca37bc4196f2fd097a957faa0d"
server_key_version: "encryptionkey@0"
I20260927 14:56:46.590124 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "38bb22782624453886cb350654207fc8"
format_stamp: "Formatted at 2026-09-27 14:56:46 on dist-test-slave-pz0h"
server_key: "8ffd9fb34779f9d8942ca1f94e080835"
server_key_iv: "42bb63ca37bc4196f2fd097a957faa0d"
server_key_version: "encryptionkey@0"
I20260927 14:56:46.594285 506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 14:56:46.597035 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:46.598361 506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:56:46.598481 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "38bb22782624453886cb350654207fc8"
format_stamp: "Formatted at 2026-09-27 14:56:46 on dist-test-slave-pz0h"
server_key: "8ffd9fb34779f9d8942ca1f94e080835"
server_key_iv: "42bb63ca37bc4196f2fd097a957faa0d"
server_key_version: "encryptionkey@0"
I20260927 14:56:46.598608 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:56:46.614554 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:56:46.617818 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:56:46.617978 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:56:46.618556 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:56:46.619437 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:56:46.619488 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:46.619529 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:56:46.619550 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:46.629187 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46513
I20260927 14:56:46.629220 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46513 every 8 connection(s)
I20260927 14:56:46.630172 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:56:46.634557 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 506
I20260927 14:56:46.634675 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:56:46.634963 392 external_mini_cluster.cc:1428] Setting key a5d7b5996d53d3f2be068bd36422221f
Sep 27 14:56:46 dist-test-slave-pz0h krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 14:56:46.645648 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:46.632098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43143 (local address 127.0.98.62:34571)
0927 14:56:46.632282 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:46.632285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:46.633164 (+ 879us) server_negotiation.cc:407] Connection header received
0927 14:56:46.634565 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:46.634568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:46.634622 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:46.634723 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:46.636221 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:46.636827 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:46.638398 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:46.638642 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:46.638734 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:46.641565 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:46.641570 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:46.641573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:46.641606 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:46.643247 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:46.643907 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:46.643912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:46.643914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:46.644009 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:46.644343 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:46.644347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:46.644349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:46.644605 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:46.644729 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:46.645244 (+ 515us) server_negotiation.cc:299] Negotiation successful
0927 14:56:46.645515 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 14:56:46.646522 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:46.632502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34571 (local address 127.0.98.1:43143)
0927 14:56:46.632995 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:46.633042 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:46.634321 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:46.634966 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:46.634977 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:46.635490 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:46.636077 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:46.636091 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:46.636975 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:46.636978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:46.638274 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:46.638285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:46.638818 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:46.638822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:46.638996 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:46.639594 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:46.639620 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:46.641358 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:46.643405 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:46.643411 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:46.643427 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:46.643799 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:46.644096 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:46.644098 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:46.644099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:46.644257 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:46.644728 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:46.644734 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:46.645015 (+ 281us) client_negotiation.cc:769] Sending connection context
0927 14:56:46.645201 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 14:56:46.645581 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 14:56:46.647811 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34571
I20260927 14:56:46.648084 635 heartbeater.cc:464] Registering TS with master...
I20260927 14:56:46.648641 635 heartbeater.cc:511] Master 127.0.98.62:34571 requested a full tablet report, sending...
I20260927 14:56:46.650347 444 ts_manager.cc:194] Registered new tserver with Master: 38bb22782624453886cb350654207fc8 (127.0.98.1:46513)
I20260927 14:56:46.651999 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43143
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:56:46.662441 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34571
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:56:46.775049 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:56:46.775334 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:56:46.775382 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:56:46.779552 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:56:46.779630 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:56:46.779699 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:56:46.785604 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34571
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:56:46.786743 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:56:46.787851 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:56:46.797664 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:46.797674 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:46.797856 640 server_base.cc:1034] running on GCE node
W20260927 14:56:46.797727 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:46.798408 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:56:46.799166 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:56:46.800355 640 hybrid_clock.cc:648] HybridClock initialized: now 1790521006800347 us; error 38 us; skew 500 ppm
Sep 27 14:56:46 dist-test-slave-pz0h krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:56:46.803534 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:56:46.804538 640 webserver.cc:460] Webserver started at http://127.0.98.2:35533/ using document root <none> and password file <none>
I20260927 14:56:46.805148 640 fs_manager.cc:359] Metadata directory not provided
I20260927 14:56:46.805217 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:56:46.805433 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:56:46.807761 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-1/data/instance:
uuid: "7c55978841dd47329f275c5b71cbea60"
format_stamp: "Formatted at 2026-09-27 14:56:46 on dist-test-slave-pz0h"
server_key: "4ddae2883017399478318f5960dc4b3c"
server_key_iv: "1d02ed8e9d7bd35befbb93962702fd78"
server_key_version: "encryptionkey@0"
I20260927 14:56:46.808396 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7c55978841dd47329f275c5b71cbea60"
format_stamp: "Formatted at 2026-09-27 14:56:46 on dist-test-slave-pz0h"
server_key: "4ddae2883017399478318f5960dc4b3c"
server_key_iv: "1d02ed8e9d7bd35befbb93962702fd78"
server_key_version: "encryptionkey@0"
I20260927 14:56:46.812829 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 14:56:46.816035 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:46.817940 640 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 14:56:46.818100 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "7c55978841dd47329f275c5b71cbea60"
format_stamp: "Formatted at 2026-09-27 14:56:46 on dist-test-slave-pz0h"
server_key: "4ddae2883017399478318f5960dc4b3c"
server_key_iv: "1d02ed8e9d7bd35befbb93962702fd78"
server_key_version: "encryptionkey@0"
I20260927 14:56:46.818246 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:56:46.830926 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:56:46.833911 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:56:46.834110 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:56:46.834753 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:56:46.835633 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:56:46.835683 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:46.835726 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:56:46.835739 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:46.844861 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40571
I20260927 14:56:46.844892 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40571 every 8 connection(s)
I20260927 14:56:46.845870 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:56:46.848794 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 640
I20260927 14:56:46.848923 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:56:46.849211 392 external_mini_cluster.cc:1428] Setting key 67f0c8a21a3d13be521ba5734af66116
Sep 27 14:56:46 dist-test-slave-pz0h krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 14:56:46.860447 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:46.847733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42937 (local address 127.0.98.62:34571)
0927 14:56:46.847890 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:46.847893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:46.848698 (+ 805us) server_negotiation.cc:407] Connection header received
0927 14:56:46.850231 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:46.850234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:46.850276 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:46.850333 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:46.851887 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:46.852422 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:46.854014 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:46.854220 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:46.854299 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:46.856986 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:46.856990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:46.856992 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:46.857020 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:46.858334 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:46.858869 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:46.858873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:46.858875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:46.858953 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:46.859292 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:46.859294 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:46.859295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:46.859482 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:46.859555 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:46.860128 (+ 573us) server_negotiation.cc:299] Negotiation successful
0927 14:56:46.860268 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 14:56:46.861385 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:46.848089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34571 (local address 127.0.98.2:42937)
0927 14:56:46.848520 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:46.848578 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:46.850022 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:46.850516 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:46.850524 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:46.851036 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:46.851735 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:46.851755 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:46.852547 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:46.852551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:46.853904 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:46.853915 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:46.854425 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:46.854429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:46.854579 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:46.855138 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:46.855154 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:46.856797 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:46.858459 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:46.858465 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:46.858483 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:46.858748 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:46.859050 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:46.859053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:46.859055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:46.859207 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:46.859559 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:46.859564 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:46.859874 (+ 310us) client_negotiation.cc:769] Sending connection context
0927 14:56:46.860036 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 14:56:46.860406 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":106,"threads_started":1}
I20260927 14:56:46.863924 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34571
I20260927 14:56:46.864208 769 heartbeater.cc:464] Registering TS with master...
I20260927 14:56:46.864833 769 heartbeater.cc:511] Master 127.0.98.62:34571 requested a full tablet report, sending...
I20260927 14:56:46.866077 444 ts_manager.cc:194] Registered new tserver with Master: 7c55978841dd47329f275c5b71cbea60 (127.0.98.2:40571)
I20260927 14:56:46.866942 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:56:46.876030 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34571
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:56:46.992086 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:56:46.992393 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:56:46.992439 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:56:46.996740 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:56:46.996821 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:56:46.996894 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:56:47.002530 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34571
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:56:47.003763 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:56:47.004877 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:56:47.015004 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:47.015097 774 server_base.cc:1034] running on GCE node
W20260927 14:56:47.015002 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:47.015002 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:47.015673 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:56:47.016291 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:56:47.017452 774 hybrid_clock.cc:648] HybridClock initialized: now 1790521007017436 us; error 30 us; skew 500 ppm
Sep 27 14:56:47 dist-test-slave-pz0h krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:56:47.020184 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:56:47.021095 774 webserver.cc:460] Webserver started at http://127.0.98.3:42043/ using document root <none> and password file <none>
I20260927 14:56:47.021628 774 fs_manager.cc:359] Metadata directory not provided
I20260927 14:56:47.021687 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:56:47.021894 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:56:47.024070 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-2/data/instance:
uuid: "407131a8f72345f9b307de2e57ce308a"
format_stamp: "Formatted at 2026-09-27 14:56:47 on dist-test-slave-pz0h"
server_key: "71cdfde7e888853f9159c27a7ce763f1"
server_key_iv: "a77ec6cd5318fa75a4006ec8a4c2cdb7"
server_key_version: "encryptionkey@0"
I20260927 14:56:47.024626 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "407131a8f72345f9b307de2e57ce308a"
format_stamp: "Formatted at 2026-09-27 14:56:47 on dist-test-slave-pz0h"
server_key: "71cdfde7e888853f9159c27a7ce763f1"
server_key_iv: "a77ec6cd5318fa75a4006ec8a4c2cdb7"
server_key_version: "encryptionkey@0"
I20260927 14:56:47.028316 774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 14:56:47.030741 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:47.032091 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:56:47.032217 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "407131a8f72345f9b307de2e57ce308a"
format_stamp: "Formatted at 2026-09-27 14:56:47 on dist-test-slave-pz0h"
server_key: "71cdfde7e888853f9159c27a7ce763f1"
server_key_iv: "a77ec6cd5318fa75a4006ec8a4c2cdb7"
server_key_version: "encryptionkey@0"
I20260927 14:56:47.032339 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:56:47.047742 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:56:47.050968 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:56:47.051151 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:56:47.051748 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:56:47.052572 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:56:47.052618 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:47.052670 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:56:47.052699 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:47.061681 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46287
I20260927 14:56:47.061728 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46287 every 8 connection(s)
I20260927 14:56:47.062637 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 14:56:47 dist-test-slave-pz0h krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:56:47.072758 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 774
I20260927 14:56:47.072880 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790521004237705-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:56:47.073127 392 external_mini_cluster.cc:1428] Setting key 5be7d7cdc2a2af15bb73e85056cd49db
I20260927 14:56:47.076051 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:47.064473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52053 (local address 127.0.98.62:34571)
0927 14:56:47.064619 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:47.064623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:47.065553 (+ 930us) server_negotiation.cc:407] Connection header received
0927 14:56:47.066846 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:47.066850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:47.066903 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:47.066976 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:47.068135 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:47.068678 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:47.069842 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:47.070057 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:47.070129 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:47.072525 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:47.072529 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:47.072532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:47.072561 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:47.074010 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:47.074496 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:47.074500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:47.074502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:47.074581 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:47.074952 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:47.074955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:47.074957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:47.075184 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:47.075279 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:47.075825 (+ 546us) server_negotiation.cc:299] Negotiation successful
0927 14:56:47.075933 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 14:56:47.076850 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:47.064906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34571 (local address 127.0.98.3:52053)
0927 14:56:47.065391 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:47.065435 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:47.066613 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:47.067134 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:47.067141 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:47.067491 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:47.067982 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:47.067996 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:47.068819 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:47.068822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:47.069718 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:47.069729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:47.070173 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:47.070175 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:47.070316 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:47.070955 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:47.070977 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:47.072364 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:47.074110 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:47.074117 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:47.074133 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:47.074403 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:47.074657 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:47.074659 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:47.074661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:47.074868 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:47.075286 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:47.075293 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:47.075563 (+ 270us) client_negotiation.cc:769] Sending connection context
0927 14:56:47.075768 (+ 205us) client_negotiation.cc:240] Negotiation successful
0927 14:56:47.076041 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":124,"threads_started":1}
I20260927 14:56:47.077855 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34571
I20260927 14:56:47.078088 903 heartbeater.cc:464] Registering TS with master...
I20260927 14:56:47.078573 903 heartbeater.cc:511] Master 127.0.98.62:34571 requested a full tablet report, sending...
I20260927 14:56:47.079629 444 ts_manager.cc:194] Registered new tserver with Master: 407131a8f72345f9b307de2e57ce308a (127.0.98.3:46287)
I20260927 14:56:47.080173 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52053
I20260927 14:56:47.087564 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 14:56:47 dist-test-slave-pz0h krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521007, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 14:56:47 dist-test-slave-pz0h krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521007, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 14:56:47.112968 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:47.103275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58200 (local address 127.0.98.1:46513)
0927 14:56:47.103508 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:47.103522 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:47.103772 (+ 250us) server_negotiation.cc:407] Connection header received
0927 14:56:47.103867 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:47.103872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:47.104058 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:47.104177 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:47.104743 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:47.105492 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:47.106294 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:47.106712 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:47.106838 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:47.109314 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:47.109324 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:47.109338 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:47.109368 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:47.111039 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:47.111528 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:47.111534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:47.111535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:47.111630 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:47.111875 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:47.111877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:47.111878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:47.112150 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:47.112337 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:47.112605 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 14:56:47.112775 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":64,"threads_started":1}
W20260927 14:56:47.113935 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58200
Sep 27 14:56:47 dist-test-slave-pz0h krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 14:56:47 dist-test-slave-pz0h krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 14:56:47.139896 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:47.131415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58214 (local address 127.0.98.1:46513)
0927 14:56:47.131564 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:47.131569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:47.131712 (+ 143us) server_negotiation.cc:407] Connection header received
0927 14:56:47.131810 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:47.131813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:47.131867 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:47.131933 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:47.132601 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:47.133150 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:47.134500 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:47.134750 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:47.134897 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:47.137221 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:47.137224 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:47.137226 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:47.137255 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:47.138544 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:47.138953 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:47.138955 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:47.138957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:47.139031 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:47.139214 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:47.139216 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:47.139217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:47.139375 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:47.139466 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:47.139637 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 14:56:47.139743 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 14:56:47.140861 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 506
I20260927 14:56:47.146725 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 640
I20260927 14:56:47.153774 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 774
I20260927 14:56:47.158747 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 412
2026-09-27T14:56:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2911 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:56:47 dist-test-slave-pz0h krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:56:47 dist-test-slave-pz0h krb5kdc[917](info): set up 2 sockets
Sep 27 14:56:47 dist-test-slave-pz0h krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260927 14:56:48.118325 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 14:56:48 dist-test-slave-pz0h krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:56:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:56:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:56:48.239337 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43669
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43669
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:56:48.353617 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:56:48.353925 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:56:48.353969 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:56:48.358179 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:56:48.358265 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:56:48.358285 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:56:48.358304 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:56:48.358317 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:56:48.364257 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:56:48.365403 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:56:48.366429 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:56:48.374766 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:48.374770 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:48.374770 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:48.375061 930 server_base.cc:1034] running on GCE node
I20260927 14:56:48.375730 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:56:48.376927 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:56:48.378130 930 hybrid_clock.cc:648] HybridClock initialized: now 1790521008378107 us; error 46 us; skew 500 ppm
Sep 27 14:56:48 dist-test-slave-pz0h krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:56:48.381102 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:56:48.382143 930 webserver.cc:460] Webserver started at http://127.0.98.62:45299/ using document root <none> and password file <none>
I20260927 14:56:48.383278 930 fs_manager.cc:359] Metadata directory not provided
I20260927 14:56:48.383419 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:56:48.383671 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:56:48.385885 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "1492f4003a004b7784be818b0b7c1c09"
format_stamp: "Formatted at 2026-09-27 14:56:48 on dist-test-slave-pz0h"
server_key: "a67acc8945faa5f5698a36447926119a"
server_key_iv: "d9dc99fb8d8101d641bfc87af5cbedf1"
server_key_version: "encryptionkey@0"
I20260927 14:56:48.386468 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "1492f4003a004b7784be818b0b7c1c09"
format_stamp: "Formatted at 2026-09-27 14:56:48 on dist-test-slave-pz0h"
server_key: "a67acc8945faa5f5698a36447926119a"
server_key_iv: "d9dc99fb8d8101d641bfc87af5cbedf1"
server_key_version: "encryptionkey@0"
I20260927 14:56:48.390714 930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 14:56:48.394095 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:48.395763 930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 14:56:48.395922 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "1492f4003a004b7784be818b0b7c1c09"
format_stamp: "Formatted at 2026-09-27 14:56:48 on dist-test-slave-pz0h"
server_key: "a67acc8945faa5f5698a36447926119a"
server_key_iv: "d9dc99fb8d8101d641bfc87af5cbedf1"
server_key_version: "encryptionkey@0"
I20260927 14:56:48.396049 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:56:48.418144 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:56:48.421422 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:56:48.421614 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:56:48.428963 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43669
I20260927 14:56:48.428982 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43669 every 8 connection(s)
I20260927 14:56:48.429955 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:56:48.433024 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:48.436250 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 930
I20260927 14:56:48.436375 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 14:56:48.436610 392 external_mini_cluster.cc:1428] Setting key 8c50e6a36fd08fdf43a01c6e530c3bb0
I20260927 14:56:48.439452 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1492f4003a004b7784be818b0b7c1c09: Bootstrap starting.
I20260927 14:56:48.442145 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1492f4003a004b7784be818b0b7c1c09: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:48.443190 998 log.cc:824] T 00000000000000000000000000000000 P 1492f4003a004b7784be818b0b7c1c09: Log is configured to *not* fsync() on all Append() calls
I20260927 14:56:48.445729 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1492f4003a004b7784be818b0b7c1c09: No bootstrap required, opened a new log
Sep 27 14:56:48 dist-test-slave-pz0h krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:56:48.448906 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1492f4003a004b7784be818b0b7c1c09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1492f4003a004b7784be818b0b7c1c09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43669 } }
I20260927 14:56:48.449157 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1492f4003a004b7784be818b0b7c1c09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:48.449214 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1492f4003a004b7784be818b0b7c1c09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1492f4003a004b7784be818b0b7c1c09, State: Initialized, Role: FOLLOWER
I20260927 14:56:48.449698 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1492f4003a004b7784be818b0b7c1c09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1492f4003a004b7784be818b0b7c1c09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43669 } }
I20260927 14:56:48.449872 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1492f4003a004b7784be818b0b7c1c09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:56:48.449945 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1492f4003a004b7784be818b0b7c1c09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:56:48.450075 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1492f4003a004b7784be818b0b7c1c09 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:48.451041 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1492f4003a004b7784be818b0b7c1c09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1492f4003a004b7784be818b0b7c1c09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43669 } }
I20260927 14:56:48.451433 998 leader_election.cc:302] T 00000000000000000000000000000000 P 1492f4003a004b7784be818b0b7c1c09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1492f4003a004b7784be818b0b7c1c09; no voters:
I20260927 14:56:48.451766 998 leader_election.cc:288] T 00000000000000000000000000000000 P 1492f4003a004b7784be818b0b7c1c09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:56:48.451877 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1492f4003a004b7784be818b0b7c1c09 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:56:48.452082 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1492f4003a004b7784be818b0b7c1c09 [term 1 LEADER]: Becoming Leader. State: Replica: 1492f4003a004b7784be818b0b7c1c09, State: Running, Role: LEADER
I20260927 14:56:48.452315 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:48.438089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39522 (local address 127.0.98.62:43669)
0927 14:56:48.438532 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:48.438548 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:48.438585 (+ 37us) server_negotiation.cc:407] Connection header received
0927 14:56:48.439287 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:48.439308 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:48.439703 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:48.440019 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:48.440885 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:48.441942 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:48.443202 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:48.443852 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:48.444035 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:48.446756 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:48.446762 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:48.446777 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:48.446843 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:48.449031 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:48.449567 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:48.449573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:48.449579 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:48.449728 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:48.450012 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:48.450016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:48.450018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:48.450486 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:48.450695 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:48.451039 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 14:56:48.451349 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":136,"threads_started":1}
I20260927 14:56:48.452488 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1492f4003a004b7784be818b0b7c1c09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1492f4003a004b7784be818b0b7c1c09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43669 } }
I20260927 14:56:48.453020 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1492f4003a004b7784be818b0b7c1c09 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:56:48.453725 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1492f4003a004b7784be818b0b7c1c09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1492f4003a004b7784be818b0b7c1c09. Latest consensus state: current_term: 1 leader_uuid: "1492f4003a004b7784be818b0b7c1c09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1492f4003a004b7784be818b0b7c1c09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43669 } } }
I20260927 14:56:48.453835 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1492f4003a004b7784be818b0b7c1c09 [sys.catalog]: This master's current role is: LEADER
I20260927 14:56:48.454841 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1492f4003a004b7784be818b0b7c1c09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1492f4003a004b7784be818b0b7c1c09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1492f4003a004b7784be818b0b7c1c09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43669 } } }
I20260927 14:56:48.454963 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1492f4003a004b7784be818b0b7c1c09 [sys.catalog]: This master's current role is: LEADER
I20260927 14:56:48.455502 1011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:56:48.458491 1011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:56:48.464804 1011 catalog_manager.cc:1347] Generated new cluster ID: e4797ba511d247e09e89127c32df6154
I20260927 14:56:48.464895 1011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:56:48.472952 1011 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:56:48.473904 1011 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:56:48.486092 1011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1492f4003a004b7784be818b0b7c1c09: Generated new TSK 0
I20260927 14:56:48.487251 1011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:56:48.521515 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43669
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:56:48.639480 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:56:48.639782 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:56:48.639844 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:56:48.644057 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:56:48.644152 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:56:48.644243 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:56:48.649860 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43669
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:56:48.651058 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:56:48.652115 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:56:48.664230 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:48.664300 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:48.664222 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:48.664458 1024 server_base.cc:1034] running on GCE node
I20260927 14:56:48.665035 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:56:48.665706 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:56:48.666863 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790521008666842 us; error 32 us; skew 500 ppm
Sep 27 14:56:48 dist-test-slave-pz0h krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:56:48.669689 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:56:48.670670 1024 webserver.cc:460] Webserver started at http://127.0.98.1:46377/ using document root <none> and password file <none>
I20260927 14:56:48.671465 1024 fs_manager.cc:359] Metadata directory not provided
I20260927 14:56:48.671666 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:56:48.671916 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:56:48.674387 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-0/data/instance:
uuid: "68a7ea9f213a4c56a2963f3af64e424f"
format_stamp: "Formatted at 2026-09-27 14:56:48 on dist-test-slave-pz0h"
server_key: "f8f77487c65a9a047d3f0d6030de7ef5"
server_key_iv: "425e1e9c905736bef36588d5e8fc24c8"
server_key_version: "encryptionkey@0"
I20260927 14:56:48.675064 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "68a7ea9f213a4c56a2963f3af64e424f"
format_stamp: "Formatted at 2026-09-27 14:56:48 on dist-test-slave-pz0h"
server_key: "f8f77487c65a9a047d3f0d6030de7ef5"
server_key_iv: "425e1e9c905736bef36588d5e8fc24c8"
server_key_version: "encryptionkey@0"
I20260927 14:56:48.679104 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 14:56:48.681892 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:48.683486 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:56:48.683636 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "68a7ea9f213a4c56a2963f3af64e424f"
format_stamp: "Formatted at 2026-09-27 14:56:48 on dist-test-slave-pz0h"
server_key: "f8f77487c65a9a047d3f0d6030de7ef5"
server_key_iv: "425e1e9c905736bef36588d5e8fc24c8"
server_key_version: "encryptionkey@0"
I20260927 14:56:48.683771 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:56:48.733989 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:56:48.737737 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:56:48.737960 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:56:48.738622 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:56:48.739559 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:56:48.739611 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:48.739667 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:56:48.739696 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:48.748968 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35591
I20260927 14:56:48.748996 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35591 every 8 connection(s)
I20260927 14:56:48.749962 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:56:48.759508 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 1024
I20260927 14:56:48.759626 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:56:48.759895 392 external_mini_cluster.cc:1428] Setting key d2dd5eadec70b02e5715274a1af454df
Sep 27 14:56:48 dist-test-slave-pz0h krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:56:48.764653 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:48.751966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59937 (local address 127.0.98.62:43669)
0927 14:56:48.752130 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:48.752134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:48.753028 (+ 894us) server_negotiation.cc:407] Connection header received
0927 14:56:48.754463 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:48.754467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:48.754527 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:48.754602 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:48.755927 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:48.756467 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:48.757649 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:48.757895 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:48.757984 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:48.760936 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:48.760940 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:48.760943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:48.760967 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:48.762444 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:48.763039 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:48.763043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:48.763044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:48.763128 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:48.763485 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:48.763488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:48.763489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:48.763680 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:48.763772 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:48.764335 (+ 563us) server_negotiation.cc:299] Negotiation successful
0927 14:56:48.764479 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 14:56:48.765556 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:48.752337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43669 (local address 127.0.98.1:59937)
0927 14:56:48.752875 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:48.752923 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:48.754232 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:48.754823 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:48.754832 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:48.755247 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:48.755765 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:48.755780 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:48.756595 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:48.756598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:48.757519 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:48.757529 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:48.758045 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:48.758050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:48.758315 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:48.758964 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:48.758982 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:48.760757 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:48.762569 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:48.762575 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:48.762590 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:48.762937 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:48.763217 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:48.763220 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:48.763222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:48.763384 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:48.763781 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:48.763789 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:48.764070 (+ 281us) client_negotiation.cc:769] Sending connection context
0927 14:56:48.764248 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 14:56:48.764612 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":123,"threads_started":1}
I20260927 14:56:48.767437 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43669
I20260927 14:56:48.767829 1153 heartbeater.cc:464] Registering TS with master...
I20260927 14:56:48.768637 1153 heartbeater.cc:511] Master 127.0.98.62:43669 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 14:56:48.770409 962 ts_manager.cc:194] Registered new tserver with Master: 68a7ea9f213a4c56a2963f3af64e424f (127.0.98.1:35591)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 14:56:48.771785 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:56:48.785458 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43669
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:56:48.902555 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:56:48.902884 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:56:48.902947 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:56:48.907191 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:56:48.907265 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:56:48.907341 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:56:48.912940 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43669
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:56:48.914108 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:56:48.915216 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:56:48.925446 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:48.925446 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:48.925459 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:48.925956 1158 server_base.cc:1034] running on GCE node
I20260927 14:56:48.926379 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:56:48.927101 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:56:48.930430 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790521008930210 us; error 227 us; skew 500 ppm
Sep 27 14:56:48 dist-test-slave-pz0h krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:56:48.933504 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:56:48.934468 1158 webserver.cc:460] Webserver started at http://127.0.98.2:35475/ using document root <none> and password file <none>
I20260927 14:56:48.935087 1158 fs_manager.cc:359] Metadata directory not provided
I20260927 14:56:48.935163 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:56:48.935377 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:56:48.937525 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-1/data/instance:
uuid: "1c13c833aa0243eca910b86612298825"
format_stamp: "Formatted at 2026-09-27 14:56:48 on dist-test-slave-pz0h"
server_key: "772112dfa25e284f777089140a75e050"
server_key_iv: "76ade7344e810a0c63ef65c1b626d3ce"
server_key_version: "encryptionkey@0"
I20260927 14:56:48.938110 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1c13c833aa0243eca910b86612298825"
format_stamp: "Formatted at 2026-09-27 14:56:48 on dist-test-slave-pz0h"
server_key: "772112dfa25e284f777089140a75e050"
server_key_iv: "76ade7344e810a0c63ef65c1b626d3ce"
server_key_version: "encryptionkey@0"
I20260927 14:56:48.942122 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 14:56:48.944801 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:48.946046 1158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 14:56:48.946166 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "1c13c833aa0243eca910b86612298825"
format_stamp: "Formatted at 2026-09-27 14:56:48 on dist-test-slave-pz0h"
server_key: "772112dfa25e284f777089140a75e050"
server_key_iv: "76ade7344e810a0c63ef65c1b626d3ce"
server_key_version: "encryptionkey@0"
I20260927 14:56:48.946287 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:56:48.955863 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:56:48.958657 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:56:48.958890 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:56:48.959511 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:56:48.960369 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:56:48.960417 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:48.960471 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:56:48.960495 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:48.968946 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44873
I20260927 14:56:48.968969 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44873 every 8 connection(s)
I20260927 14:56:48.969880 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:56:48.972482 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 1158
I20260927 14:56:48.972589 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:56:48.972831 392 external_mini_cluster.cc:1428] Setting key 5d0b38f5887402655d5aa33e205fca7a
Sep 27 14:56:48 dist-test-slave-pz0h krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 14:56:48.984593 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:48.971872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46507 (local address 127.0.98.62:43669)
0927 14:56:48.972037 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:48.972041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:48.972892 (+ 851us) server_negotiation.cc:407] Connection header received
0927 14:56:48.974464 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:48.974468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:48.974519 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:48.974589 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:48.975987 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:48.976506 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:48.977708 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:48.977933 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:48.978022 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:48.980972 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:48.980976 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:48.980978 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:48.981006 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:48.982287 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:48.982834 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:48.982837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:48.982839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:48.982919 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:48.983294 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:48.983296 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:48.983297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:48.983490 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:48.983579 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:48.984298 (+ 719us) server_negotiation.cc:299] Negotiation successful
0927 14:56:48.984427 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 14:56:48.985438 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:48.972237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43669 (local address 127.0.98.2:46507)
0927 14:56:48.972730 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:48.972785 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:48.974241 (+ 1456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:48.974822 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:48.974834 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:48.975294 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:48.975833 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:48.975848 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:48.976633 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:48.976637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:48.977578 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:48.977591 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:48.978082 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:48.978086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:48.978261 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:48.978963 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:48.978980 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:48.980763 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:48.982407 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:48.982413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:48.982428 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:48.982719 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:48.983025 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:48.983028 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:48.983029 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:48.983218 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:48.983591 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:48.983599 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:48.983983 (+ 384us) client_negotiation.cc:769] Sending connection context
0927 14:56:48.984185 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 14:56:48.984478 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":123,"threads_started":1}
I20260927 14:56:48.986660 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43669
I20260927 14:56:48.986969 1287 heartbeater.cc:464] Registering TS with master...
I20260927 14:56:48.987619 1287 heartbeater.cc:511] Master 127.0.98.62:43669 requested a full tablet report, sending...
I20260927 14:56:48.988821 962 ts_manager.cc:194] Registered new tserver with Master: 1c13c833aa0243eca910b86612298825 (127.0.98.2:44873)
I20260927 14:56:48.989390 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:56:48.999219 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43669
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:56:49.108942 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:56:49.109254 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:56:49.109316 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:56:49.113508 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:56:49.113595 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:56:49.113682 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:56:49.119264 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43669
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:56:49.120383 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:56:49.121404 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:56:49.131081 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:49.131084 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:49.131129 1292 server_base.cc:1034] running on GCE node
W20260927 14:56:49.131081 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:49.131769 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:56:49.132442 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:56:49.133620 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790521009133598 us; error 36 us; skew 500 ppm
Sep 27 14:56:49 dist-test-slave-pz0h krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:56:49.136404 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:56:49.137347 1292 webserver.cc:460] Webserver started at http://127.0.98.3:44869/ using document root <none> and password file <none>
I20260927 14:56:49.137893 1292 fs_manager.cc:359] Metadata directory not provided
I20260927 14:56:49.137960 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:56:49.138150 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:56:49.140246 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-2/data/instance:
uuid: "f9897b5ed7904c748b623380b6228d4b"
format_stamp: "Formatted at 2026-09-27 14:56:49 on dist-test-slave-pz0h"
server_key: "b6bcbb17bcc0aa799241ddd44f077c8f"
server_key_iv: "597ac9aa947a2fd1ce16ffdcef43310b"
server_key_version: "encryptionkey@0"
I20260927 14:56:49.140821 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f9897b5ed7904c748b623380b6228d4b"
format_stamp: "Formatted at 2026-09-27 14:56:49 on dist-test-slave-pz0h"
server_key: "b6bcbb17bcc0aa799241ddd44f077c8f"
server_key_iv: "597ac9aa947a2fd1ce16ffdcef43310b"
server_key_version: "encryptionkey@0"
I20260927 14:56:49.145036 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 14:56:49.147605 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:49.148931 1292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:56:49.149065 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "f9897b5ed7904c748b623380b6228d4b"
format_stamp: "Formatted at 2026-09-27 14:56:49 on dist-test-slave-pz0h"
server_key: "b6bcbb17bcc0aa799241ddd44f077c8f"
server_key_iv: "597ac9aa947a2fd1ce16ffdcef43310b"
server_key_version: "encryptionkey@0"
I20260927 14:56:49.149215 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:56:49.161340 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:56:49.164532 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:56:49.164723 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:56:49.165357 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:56:49.166239 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:56:49.166287 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:49.166342 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:56:49.166368 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:49.174996 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38233
I20260927 14:56:49.175009 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38233 every 8 connection(s)
I20260927 14:56:49.175971 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 14:56:49 dist-test-slave-pz0h krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:56:49.185847 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 1292
I20260927 14:56:49.185948 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790521004237705-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:56:49.186197 392 external_mini_cluster.cc:1428] Setting key 9c96913d96ea8053b86bf7fe652d56a5
I20260927 14:56:49.190267 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:49.177734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35825 (local address 127.0.98.62:43669)
0927 14:56:49.177891 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:49.177896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:49.178787 (+ 891us) server_negotiation.cc:407] Connection header received
0927 14:56:49.180173 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:49.180176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:49.180251 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:49.180327 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:49.181538 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.182065 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.183294 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.183528 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.183625 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:49.186406 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:49.186410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:49.186412 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:49.186442 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:49.188151 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:49.188763 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:49.188766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:49.188767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:49.188853 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:49.189226 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:49.189228 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:49.189229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:49.189424 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:49.189499 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:49.190031 (+ 532us) server_negotiation.cc:299] Negotiation successful
0927 14:56:49.190144 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 14:56:49.191175 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:49.178152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43669 (local address 127.0.98.3:35825)
0927 14:56:49.178610 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:49.178655 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:49.179932 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:49.180497 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:49.180505 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:49.180885 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:49.181403 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:49.181417 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.182177 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.182179 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:49.183133 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:49.183143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.183671 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.183674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:49.183839 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:49.184474 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:49.184492 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:49.186169 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:49.188324 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:49.188332 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:49.188352 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:49.188663 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:49.188964 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:49.188967 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:49.188969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:49.189151 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:49.189502 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:49.189506 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:49.189783 (+ 277us) client_negotiation.cc:769] Sending connection context
0927 14:56:49.189950 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 14:56:49.190229 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":119,"threads_started":1}
I20260927 14:56:49.192417 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43669
I20260927 14:56:49.192667 1421 heartbeater.cc:464] Registering TS with master...
I20260927 14:56:49.193176 1421 heartbeater.cc:511] Master 127.0.98.62:43669 requested a full tablet report, sending...
I20260927 14:56:49.194365 962 ts_manager.cc:194] Registered new tserver with Master: f9897b5ed7904c748b623380b6228d4b (127.0.98.3:38233)
I20260927 14:56:49.195000 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35825
I20260927 14:56:49.200651 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:56:49.211964 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:49.204448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41192 (local address 127.0.98.62:43669)
0927 14:56:49.204615 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:49.204619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:49.204687 (+ 68us) server_negotiation.cc:407] Connection header received
0927 14:56:49.204787 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:49.204789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:49.204833 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:49.204904 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:49.205566 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.206077 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.206932 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.207149 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.207240 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:49.208598 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:49.208601 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:49.208603 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:49.208631 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:49.210242 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:49.210692 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:49.210695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:49.210696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:49.210777 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:49.211053 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:49.211055 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:49.211056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:49.211292 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:49.211395 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:49.211691 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 14:56:49.211803 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 14:56:49.217020 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:56:49.220188 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:56:49.234776 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:49.229427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58138 (local address 127.0.98.3:38233)
0927 14:56:49.229896 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:49.229908 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:49.229931 (+ 23us) server_negotiation.cc:407] Connection header received
0927 14:56:49.230004 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:49.230010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:49.230230 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:49.230386 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:56:49.230699 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.231688 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.233179 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.234099 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.234197 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:49.234356 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:49.234521 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 14:56:49.234615 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":122,"threads_started":1}
I20260927 14:56:49.234769 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:49.229265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38233 (local address 127.0.0.1:58138)
0927 14:56:49.229645 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:49.229697 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:49.229831 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:49.230393 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:49.230399 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:49.230423 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:56:49.230565 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:49.230572 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.231835 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.231838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:49.233065 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:49.233074 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.234198 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.234201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:49.234331 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:49.234370 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 14:56:49.234474 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 14:56:49.234605 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":110,"threads_started":1}
I20260927 14:56:49.235555 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:49.229517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44873 (local address 127.0.0.1:35326)
0927 14:56:49.230140 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:49.230154 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:49.230262 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:49.231323 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:49.231326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:49.231342 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:56:49.231437 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:49.231442 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.232772 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.232775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:49.234121 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:49.234128 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.235155 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.235159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:49.235259 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:49.235284 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 14:56:49.235353 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 14:56:49.235425 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":1,"thread_start_us":100,"threads_started":1}
I20260927 14:56:49.235641 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:49.230225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35326 (local address 127.0.98.2:44873)
0927 14:56:49.230711 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:49.230722 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:49.230745 (+ 23us) server_negotiation.cc:407] Connection header received
0927 14:56:49.230841 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:49.230846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:49.231035 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:49.231175 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:56:49.231551 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.232649 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.234252 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.235035 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.235136 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:49.235279 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:49.235396 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 14:56:49.235491 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":87,"threads_started":1}
I20260927 14:56:49.236137 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:49.230065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35591 (local address 127.0.0.1:52712)
0927 14:56:49.231075 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:49.231090 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:49.231176 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:49.231630 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:49.231633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:49.231650 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:56:49.231750 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:49.231756 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.232693 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.232696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:49.233887 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:49.233896 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.235473 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.235476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:49.235571 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:49.235593 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 14:56:49.235941 (+ 348us) client_negotiation.cc:240] Negotiation successful
0927 14:56:49.236017 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":954,"mutex_wait_us":34,"thread_start_us":91,"threads_started":1}
I20260927 14:56:49.236232 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:49.230465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52712 (local address 127.0.98.1:35591)
0927 14:56:49.230843 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:49.230856 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:49.231262 (+ 406us) server_negotiation.cc:407] Connection header received
0927 14:56:49.231339 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:49.231343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:49.231511 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:49.231639 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:56:49.231851 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.232564 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.234013 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.235194 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.235286 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:49.235415 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:49.235982 (+ 567us) server_negotiation.cc:299] Negotiation successful
0927 14:56:49.236085 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":64,"threads_started":1}
I20260927 14:56:49.238175 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 4a951ad712f94afe89ce8b76b03b4919 (DEFAULT_TABLE table=test-table [id=dd5598c2cafa46d4abc0ec3bf820da47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:56:49.238168 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 4a951ad712f94afe89ce8b76b03b4919 (DEFAULT_TABLE table=test-table [id=dd5598c2cafa46d4abc0ec3bf820da47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:56:49.239190 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a951ad712f94afe89ce8b76b03b4919. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:49.239202 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a951ad712f94afe89ce8b76b03b4919. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:49.238919 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 4a951ad712f94afe89ce8b76b03b4919 (DEFAULT_TABLE table=test-table [id=dd5598c2cafa46d4abc0ec3bf820da47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:56:49.239835 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a951ad712f94afe89ce8b76b03b4919. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:49.246284 1438 tablet_bootstrap.cc:492] T 4a951ad712f94afe89ce8b76b03b4919 P 1c13c833aa0243eca910b86612298825: Bootstrap starting.
I20260927 14:56:49.246694 1439 tablet_bootstrap.cc:492] T 4a951ad712f94afe89ce8b76b03b4919 P f9897b5ed7904c748b623380b6228d4b: Bootstrap starting.
I20260927 14:56:49.248808 1439 tablet_bootstrap.cc:654] T 4a951ad712f94afe89ce8b76b03b4919 P f9897b5ed7904c748b623380b6228d4b: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:49.249248 1438 tablet_bootstrap.cc:654] T 4a951ad712f94afe89ce8b76b03b4919 P 1c13c833aa0243eca910b86612298825: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:49.250165 1439 log.cc:824] T 4a951ad712f94afe89ce8b76b03b4919 P f9897b5ed7904c748b623380b6228d4b: Log is configured to *not* fsync() on all Append() calls
I20260927 14:56:49.250156 1438 log.cc:824] T 4a951ad712f94afe89ce8b76b03b4919 P 1c13c833aa0243eca910b86612298825: Log is configured to *not* fsync() on all Append() calls
I20260927 14:56:49.250923 1440 tablet_bootstrap.cc:492] T 4a951ad712f94afe89ce8b76b03b4919 P 68a7ea9f213a4c56a2963f3af64e424f: Bootstrap starting.
I20260927 14:56:49.252593 1438 tablet_bootstrap.cc:492] T 4a951ad712f94afe89ce8b76b03b4919 P 1c13c833aa0243eca910b86612298825: No bootstrap required, opened a new log
I20260927 14:56:49.252848 1438 ts_tablet_manager.cc:1397] T 4a951ad712f94afe89ce8b76b03b4919 P 1c13c833aa0243eca910b86612298825: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 14:56:49.253068 1439 tablet_bootstrap.cc:492] T 4a951ad712f94afe89ce8b76b03b4919 P f9897b5ed7904c748b623380b6228d4b: No bootstrap required, opened a new log
I20260927 14:56:49.253299 1439 ts_tablet_manager.cc:1397] T 4a951ad712f94afe89ce8b76b03b4919 P f9897b5ed7904c748b623380b6228d4b: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 14:56:49.254033 1440 tablet_bootstrap.cc:654] T 4a951ad712f94afe89ce8b76b03b4919 P 68a7ea9f213a4c56a2963f3af64e424f: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:49.255445 1440 log.cc:824] T 4a951ad712f94afe89ce8b76b03b4919 P 68a7ea9f213a4c56a2963f3af64e424f: Log is configured to *not* fsync() on all Append() calls
I20260927 14:56:49.256321 1438 raft_consensus.cc:348] T 4a951ad712f94afe89ce8b76b03b4919 P 1c13c833aa0243eca910b86612298825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9897b5ed7904c748b623380b6228d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38233 } } peers { permanent_uuid: "68a7ea9f213a4c56a2963f3af64e424f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35591 } } peers { permanent_uuid: "1c13c833aa0243eca910b86612298825" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44873 } }
I20260927 14:56:49.256565 1438 raft_consensus.cc:374] T 4a951ad712f94afe89ce8b76b03b4919 P 1c13c833aa0243eca910b86612298825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:49.256615 1438 raft_consensus.cc:729] T 4a951ad712f94afe89ce8b76b03b4919 P 1c13c833aa0243eca910b86612298825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c13c833aa0243eca910b86612298825, State: Initialized, Role: FOLLOWER
I20260927 14:56:49.257042 1438 consensus_queue.cc:260] T 4a951ad712f94afe89ce8b76b03b4919 P 1c13c833aa0243eca910b86612298825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9897b5ed7904c748b623380b6228d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38233 } } peers { permanent_uuid: "68a7ea9f213a4c56a2963f3af64e424f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35591 } } peers { permanent_uuid: "1c13c833aa0243eca910b86612298825" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44873 } }
I20260927 14:56:49.257088 1439 raft_consensus.cc:348] T 4a951ad712f94afe89ce8b76b03b4919 P f9897b5ed7904c748b623380b6228d4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9897b5ed7904c748b623380b6228d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38233 } } peers { permanent_uuid: "68a7ea9f213a4c56a2963f3af64e424f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35591 } } peers { permanent_uuid: "1c13c833aa0243eca910b86612298825" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44873 } }
I20260927 14:56:49.257364 1439 raft_consensus.cc:374] T 4a951ad712f94afe89ce8b76b03b4919 P f9897b5ed7904c748b623380b6228d4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:49.257431 1439 raft_consensus.cc:729] T 4a951ad712f94afe89ce8b76b03b4919 P f9897b5ed7904c748b623380b6228d4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9897b5ed7904c748b623380b6228d4b, State: Initialized, Role: FOLLOWER
I20260927 14:56:49.257697 1440 tablet_bootstrap.cc:492] T 4a951ad712f94afe89ce8b76b03b4919 P 68a7ea9f213a4c56a2963f3af64e424f: No bootstrap required, opened a new log
I20260927 14:56:49.257810 1287 heartbeater.cc:503] Master 127.0.98.62:43669 was elected leader, sending a full tablet report...
I20260927 14:56:49.257894 1440 ts_tablet_manager.cc:1397] T 4a951ad712f94afe89ce8b76b03b4919 P 68a7ea9f213a4c56a2963f3af64e424f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 14:56:49.257912 1439 consensus_queue.cc:260] T 4a951ad712f94afe89ce8b76b03b4919 P f9897b5ed7904c748b623380b6228d4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9897b5ed7904c748b623380b6228d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38233 } } peers { permanent_uuid: "68a7ea9f213a4c56a2963f3af64e424f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35591 } } peers { permanent_uuid: "1c13c833aa0243eca910b86612298825" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44873 } }
I20260927 14:56:49.258937 1439 ts_tablet_manager.cc:1428] T 4a951ad712f94afe89ce8b76b03b4919 P f9897b5ed7904c748b623380b6228d4b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 14:56:49.259258 1421 heartbeater.cc:503] Master 127.0.98.62:43669 was elected leader, sending a full tablet report...
I20260927 14:56:49.260834 1438 ts_tablet_manager.cc:1428] T 4a951ad712f94afe89ce8b76b03b4919 P 1c13c833aa0243eca910b86612298825: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260927 14:56:49.261703 1440 raft_consensus.cc:348] T 4a951ad712f94afe89ce8b76b03b4919 P 68a7ea9f213a4c56a2963f3af64e424f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9897b5ed7904c748b623380b6228d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38233 } } peers { permanent_uuid: "68a7ea9f213a4c56a2963f3af64e424f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35591 } } peers { permanent_uuid: "1c13c833aa0243eca910b86612298825" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44873 } }
I20260927 14:56:49.262005 1440 raft_consensus.cc:374] T 4a951ad712f94afe89ce8b76b03b4919 P 68a7ea9f213a4c56a2963f3af64e424f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:49.262068 1440 raft_consensus.cc:729] T 4a951ad712f94afe89ce8b76b03b4919 P 68a7ea9f213a4c56a2963f3af64e424f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a7ea9f213a4c56a2963f3af64e424f, State: Initialized, Role: FOLLOWER
I20260927 14:56:49.262530 1440 consensus_queue.cc:260] T 4a951ad712f94afe89ce8b76b03b4919 P 68a7ea9f213a4c56a2963f3af64e424f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9897b5ed7904c748b623380b6228d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38233 } } peers { permanent_uuid: "68a7ea9f213a4c56a2963f3af64e424f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35591 } } peers { permanent_uuid: "1c13c833aa0243eca910b86612298825" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44873 } }
I20260927 14:56:49.263237 1153 heartbeater.cc:503] Master 127.0.98.62:43669 was elected leader, sending a full tablet report...
I20260927 14:56:49.263501 1440 ts_tablet_manager.cc:1428] T 4a951ad712f94afe89ce8b76b03b4919 P 68a7ea9f213a4c56a2963f3af64e424f: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260927 14:56:49.346252 1446 raft_consensus.cc:482] T 4a951ad712f94afe89ce8b76b03b4919 P 68a7ea9f213a4c56a2963f3af64e424f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:56:49.346470 1446 raft_consensus.cc:504] T 4a951ad712f94afe89ce8b76b03b4919 P 68a7ea9f213a4c56a2963f3af64e424f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9897b5ed7904c748b623380b6228d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38233 } } peers { permanent_uuid: "68a7ea9f213a4c56a2963f3af64e424f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35591 } } peers { permanent_uuid: "1c13c833aa0243eca910b86612298825" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44873 } }
I20260927 14:56:49.347693 1446 leader_election.cc:288] T 4a951ad712f94afe89ce8b76b03b4919 P 68a7ea9f213a4c56a2963f3af64e424f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9897b5ed7904c748b623380b6228d4b (127.0.98.3:38233), 1c13c833aa0243eca910b86612298825 (127.0.98.2:44873)
I20260927 14:56:49.352319 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:49.348046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35451 (local address 127.0.98.3:38233)
0927 14:56:49.348191 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:49.348213 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:49.348407 (+ 194us) server_negotiation.cc:407] Connection header received
0927 14:56:49.348536 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:49.348539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:49.348582 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:49.348643 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:56:49.348896 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.349433 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.351197 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.351732 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.351834 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:49.351892 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:49.352127 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 14:56:49.352206 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 14:56:49.352340 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:49.347949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38233 (local address 127.0.98.1:35451)
0927 14:56:49.348309 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:49.348322 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:49.348434 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:49.348663 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:49.348666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:49.348680 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:56:49.348797 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:49.348802 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.349533 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.349536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:49.351062 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:49.351075 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.351875 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.351878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:49.351962 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:49.351985 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 14:56:49.352071 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 14:56:49.352157 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":71,"thread_start_us":95,"threads_started":1}
I20260927 14:56:49.352931 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:49.347988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44873 (local address 127.0.98.1:34215)
0927 14:56:49.348343 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:49.348354 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:49.348465 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:49.348702 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:49.348704 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:49.348719 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:56:49.348821 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:49.348826 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.349559 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.349562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:49.351062 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:49.351075 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.352537 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.352541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:49.352641 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:49.352662 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 14:56:49.352722 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 14:56:49.352797 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":50,"thread_start_us":134,"threads_started":1}
I20260927 14:56:49.352958 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:49.348114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34215 (local address 127.0.98.2:44873)
0927 14:56:49.348257 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:49.348262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:49.348452 (+ 190us) server_negotiation.cc:407] Connection header received
0927 14:56:49.348572 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:49.348575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:49.348618 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:49.348678 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:56:49.348923 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.349443 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.351948 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.352365 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.352466 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:49.352532 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:49.352761 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 14:56:49.352840 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 14:56:49.353133 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a951ad712f94afe89ce8b76b03b4919" candidate_uuid: "68a7ea9f213a4c56a2963f3af64e424f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9897b5ed7904c748b623380b6228d4b" is_pre_election: true
I20260927 14:56:49.353511 1375 raft_consensus.cc:2454] T 4a951ad712f94afe89ce8b76b03b4919 P f9897b5ed7904c748b623380b6228d4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a7ea9f213a4c56a2963f3af64e424f in term 0.
I20260927 14:56:49.353524 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a951ad712f94afe89ce8b76b03b4919" candidate_uuid: "68a7ea9f213a4c56a2963f3af64e424f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c13c833aa0243eca910b86612298825" is_pre_election: true
I20260927 14:56:49.353817 1241 raft_consensus.cc:2454] T 4a951ad712f94afe89ce8b76b03b4919 P 1c13c833aa0243eca910b86612298825 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68a7ea9f213a4c56a2963f3af64e424f in term 0.
I20260927 14:56:49.354043 1041 leader_election.cc:302] T 4a951ad712f94afe89ce8b76b03b4919 P 68a7ea9f213a4c56a2963f3af64e424f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a7ea9f213a4c56a2963f3af64e424f, f9897b5ed7904c748b623380b6228d4b; no voters:
I20260927 14:56:49.354529 1446 raft_consensus.cc:2781] T 4a951ad712f94afe89ce8b76b03b4919 P 68a7ea9f213a4c56a2963f3af64e424f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:56:49.354597 1446 raft_consensus.cc:482] T 4a951ad712f94afe89ce8b76b03b4919 P 68a7ea9f213a4c56a2963f3af64e424f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:56:49.354630 1446 raft_consensus.cc:3037] T 4a951ad712f94afe89ce8b76b03b4919 P 68a7ea9f213a4c56a2963f3af64e424f [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:49.355713 1446 raft_consensus.cc:504] T 4a951ad712f94afe89ce8b76b03b4919 P 68a7ea9f213a4c56a2963f3af64e424f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9897b5ed7904c748b623380b6228d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38233 } } peers { permanent_uuid: "68a7ea9f213a4c56a2963f3af64e424f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35591 } } peers { permanent_uuid: "1c13c833aa0243eca910b86612298825" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44873 } }
I20260927 14:56:49.356046 1446 leader_election.cc:288] T 4a951ad712f94afe89ce8b76b03b4919 P 68a7ea9f213a4c56a2963f3af64e424f [CANDIDATE]: Term 1 election: Requested vote from peers f9897b5ed7904c748b623380b6228d4b (127.0.98.3:38233), 1c13c833aa0243eca910b86612298825 (127.0.98.2:44873)
I20260927 14:56:49.356464 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a951ad712f94afe89ce8b76b03b4919" candidate_uuid: "68a7ea9f213a4c56a2963f3af64e424f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c13c833aa0243eca910b86612298825"
I20260927 14:56:49.356482 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a951ad712f94afe89ce8b76b03b4919" candidate_uuid: "68a7ea9f213a4c56a2963f3af64e424f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9897b5ed7904c748b623380b6228d4b"
I20260927 14:56:49.356580 1241 raft_consensus.cc:3037] T 4a951ad712f94afe89ce8b76b03b4919 P 1c13c833aa0243eca910b86612298825 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:49.356594 1375 raft_consensus.cc:3037] T 4a951ad712f94afe89ce8b76b03b4919 P f9897b5ed7904c748b623380b6228d4b [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:49.357627 1375 raft_consensus.cc:2454] T 4a951ad712f94afe89ce8b76b03b4919 P f9897b5ed7904c748b623380b6228d4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a7ea9f213a4c56a2963f3af64e424f in term 1.
I20260927 14:56:49.357628 1241 raft_consensus.cc:2454] T 4a951ad712f94afe89ce8b76b03b4919 P 1c13c833aa0243eca910b86612298825 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68a7ea9f213a4c56a2963f3af64e424f in term 1.
I20260927 14:56:49.357944 1040 leader_election.cc:302] T 4a951ad712f94afe89ce8b76b03b4919 P 68a7ea9f213a4c56a2963f3af64e424f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c13c833aa0243eca910b86612298825, 68a7ea9f213a4c56a2963f3af64e424f; no voters:
I20260927 14:56:49.358109 1446 raft_consensus.cc:2781] T 4a951ad712f94afe89ce8b76b03b4919 P 68a7ea9f213a4c56a2963f3af64e424f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:56:49.358336 1446 raft_consensus.cc:686] T 4a951ad712f94afe89ce8b76b03b4919 P 68a7ea9f213a4c56a2963f3af64e424f [term 1 LEADER]: Becoming Leader. State: Replica: 68a7ea9f213a4c56a2963f3af64e424f, State: Running, Role: LEADER
I20260927 14:56:49.358661 1446 consensus_queue.cc:237] T 4a951ad712f94afe89ce8b76b03b4919 P 68a7ea9f213a4c56a2963f3af64e424f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9897b5ed7904c748b623380b6228d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38233 } } peers { permanent_uuid: "68a7ea9f213a4c56a2963f3af64e424f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35591 } } peers { permanent_uuid: "1c13c833aa0243eca910b86612298825" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44873 } }
I20260927 14:56:49.362126 962 catalog_manager.cc:5495] T 4a951ad712f94afe89ce8b76b03b4919 P 68a7ea9f213a4c56a2963f3af64e424f reported cstate change: term changed from 0 to 1, leader changed from <none> to 68a7ea9f213a4c56a2963f3af64e424f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "68a7ea9f213a4c56a2963f3af64e424f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9897b5ed7904c748b623380b6228d4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68a7ea9f213a4c56a2963f3af64e424f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c13c833aa0243eca910b86612298825" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44873 } health_report { overall_health: UNKNOWN } } }
Sep 27 14:56:49 dist-test-slave-pz0h krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 14:56:49.391321 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:49.381366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52726 (local address 127.0.98.1:35591)
0927 14:56:49.381494 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:49.381498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:49.382091 (+ 593us) server_negotiation.cc:407] Connection header received
0927 14:56:49.382269 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:49.382272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:49.382325 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:49.382400 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:49.383201 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.383726 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.384620 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.384843 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.384938 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:49.387557 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:49.387575 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:49.387597 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:49.387643 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:49.389452 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:49.389932 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:49.389938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:49.389939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:49.390050 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:49.390294 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:49.390298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:49.390299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:49.390575 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:49.390744 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:49.391024 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 14:56:49.391136 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 14:56:49 dist-test-slave-pz0h krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521009, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 14:56:49 dist-test-slave-pz0h krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521009, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 14:56:49.419597 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:49.410645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52734 (local address 127.0.98.1:35591)
0927 14:56:49.410859 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:49.410865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:49.411058 (+ 193us) server_negotiation.cc:407] Connection header received
0927 14:56:49.411186 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:49.411188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:49.411232 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:49.411294 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:49.411973 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.412610 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.413508 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.413741 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.413833 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:49.416403 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:49.416406 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:49.416409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:49.416438 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:49.417756 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:49.418183 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:49.418185 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:49.418186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:49.418268 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:49.418540 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:49.418564 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:49.418565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:49.418763 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:49.418931 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:49.419303 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 14:56:49.419433 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 14:56:49.426997 1422 tablet.cc:2369] T 4a951ad712f94afe89ce8b76b03b4919 P f9897b5ed7904c748b623380b6228d4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 14:56:49 dist-test-slave-pz0h krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 14:56:49 dist-test-slave-pz0h krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 14:56:49.448958 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:49.436695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52748 (local address 127.0.98.1:35591)
0927 14:56:49.436856 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:49.436861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:49.436945 (+ 84us) server_negotiation.cc:407] Connection header received
0927 14:56:49.437100 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:49.437104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:49.437145 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:49.437208 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:49.438109 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.438896 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.440101 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:49.440473 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:49.440582 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:49.445710 (+ 5128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:49.445715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:49.445718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:49.445757 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:49.447213 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:49.447734 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:49.447737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:49.447738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:49.447820 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:49.448061 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:49.448063 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:49.448064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:49.448234 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:49.448360 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:49.448541 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 14:56:49.448646 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 14:56:49.450371 1067 tablet_service.cc:2908] Rejecting scan request for tablet 4a951ad712f94afe89ce8b76b03b4919: Uninitialized: clean time has not yet been initialized
I20260927 14:56:49.452591 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 1024
I20260927 14:56:49.457934 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 1158
I20260927 14:56:49.462877 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 1292
I20260927 14:56:49.467780 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 930
2026-09-27T14:56:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2309 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:56:49 dist-test-slave-pz0h krb5kdc[1463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:56:49 dist-test-slave-pz0h krb5kdc[1463](info): set up 2 sockets
Sep 27 14:56:49 dist-test-slave-pz0h krb5kdc[1463](info): commencing operation
krb5kdc: starting...
W20260927 14:56:50.439539 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 14:56:50 dist-test-slave-pz0h krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:56:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:56:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:56:50.745563 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33685
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:56:50.856724 1476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:56:50.857023 1476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:56:50.857084 1476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:56:50.861203 1476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:56:50.861271 1476 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 14:56:50.861289 1476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:56:50.861305 1476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:56:50.861320 1476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:56:50.861334 1476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:56:50.866896 1476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33685
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:56:50.868001 1476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:56:50.868996 1476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:56:50.877395 1482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:50.877393 1481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:50.877688 1476 server_base.cc:1034] running on GCE node
W20260927 14:56:50.877393 1484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:50.878410 1476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:56:50.879674 1476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:56:50.880851 1476 hybrid_clock.cc:648] HybridClock initialized: now 1790521010880822 us; error 42 us; skew 500 ppm
Sep 27 14:56:50 dist-test-slave-pz0h krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:56:50.888700 1476 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:56:50.889850 1476 webserver.cc:460] Webserver started at http://127.0.98.62:46455/ using document root <none> and password file <none>
I20260927 14:56:50.890573 1476 fs_manager.cc:359] Metadata directory not provided
I20260927 14:56:50.890663 1476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:56:50.890941 1476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:56:50.893750 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "a4ce5e5dceed4f60bbfaa9f69fe75ad6"
format_stamp: "Formatted at 2026-09-27 14:56:50 on dist-test-slave-pz0h"
server_key: "e983cde7a7a3e164bfc7c71e270dd538"
server_key_iv: "69780f3b6032ec1641d622e1305ef5a5"
server_key_version: "encryptionkey@0"
I20260927 14:56:50.894395 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "a4ce5e5dceed4f60bbfaa9f69fe75ad6"
format_stamp: "Formatted at 2026-09-27 14:56:50 on dist-test-slave-pz0h"
server_key: "e983cde7a7a3e164bfc7c71e270dd538"
server_key_iv: "69780f3b6032ec1641d622e1305ef5a5"
server_key_version: "encryptionkey@0"
I20260927 14:56:50.898785 1476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 14:56:50.901540 1491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:50.902977 1476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:56:50.903110 1476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "a4ce5e5dceed4f60bbfaa9f69fe75ad6"
format_stamp: "Formatted at 2026-09-27 14:56:50 on dist-test-slave-pz0h"
server_key: "e983cde7a7a3e164bfc7c71e270dd538"
server_key_iv: "69780f3b6032ec1641d622e1305ef5a5"
server_key_version: "encryptionkey@0"
I20260927 14:56:50.903229 1476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:56:50.924512 1476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:56:50.927857 1476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:56:50.928059 1476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:56:50.935958 1476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33685
I20260927 14:56:50.936012 1553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33685 every 8 connection(s)
I20260927 14:56:50.936980 1476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:56:50.939946 1554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:50.942024 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 1476
I20260927 14:56:50.942126 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 14:56:50.942370 392 external_mini_cluster.cc:1428] Setting key c3a9e7cd8d89cb4e95eded340d27ff12
I20260927 14:56:50.946215 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4ce5e5dceed4f60bbfaa9f69fe75ad6: Bootstrap starting.
I20260927 14:56:50.948634 1554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4ce5e5dceed4f60bbfaa9f69fe75ad6: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:50.949376 1554 log.cc:824] T 00000000000000000000000000000000 P a4ce5e5dceed4f60bbfaa9f69fe75ad6: Log is configured to *not* fsync() on all Append() calls
Sep 27 14:56:50 dist-test-slave-pz0h krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:56:50.951608 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4ce5e5dceed4f60bbfaa9f69fe75ad6: No bootstrap required, opened a new log
I20260927 14:56:50.954991 1554 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4ce5e5dceed4f60bbfaa9f69fe75ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ce5e5dceed4f60bbfaa9f69fe75ad6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33685 } }
I20260927 14:56:50.955240 1554 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4ce5e5dceed4f60bbfaa9f69fe75ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:50.955300 1554 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4ce5e5dceed4f60bbfaa9f69fe75ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4ce5e5dceed4f60bbfaa9f69fe75ad6, State: Initialized, Role: FOLLOWER
I20260927 14:56:50.955789 1554 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4ce5e5dceed4f60bbfaa9f69fe75ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ce5e5dceed4f60bbfaa9f69fe75ad6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33685 } }
I20260927 14:56:50.955940 1554 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4ce5e5dceed4f60bbfaa9f69fe75ad6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:56:50.956005 1554 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4ce5e5dceed4f60bbfaa9f69fe75ad6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:56:50.956099 1554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4ce5e5dceed4f60bbfaa9f69fe75ad6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:50.956622 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:50.943885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33444 (local address 127.0.98.62:33685)
0927 14:56:50.944353 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:50.944369 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:50.944405 (+ 36us) server_negotiation.cc:407] Connection header received
0927 14:56:50.945295 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:50.945317 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:50.945676 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:50.946010 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:50.946855 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:50.947873 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:50.948725 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:50.949266 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:50.949427 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:50.952028 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:50.952034 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:50.952049 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:50.952077 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:50.953641 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:50.954065 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:50.954070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:50.954074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:50.954186 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:50.954452 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:50.954455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:50.954457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:50.954922 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:50.955122 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:50.955469 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 14:56:50.955765 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":156,"threads_started":1}
I20260927 14:56:50.957085 1554 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4ce5e5dceed4f60bbfaa9f69fe75ad6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ce5e5dceed4f60bbfaa9f69fe75ad6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33685 } }
I20260927 14:56:50.957415 1554 leader_election.cc:302] T 00000000000000000000000000000000 P a4ce5e5dceed4f60bbfaa9f69fe75ad6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4ce5e5dceed4f60bbfaa9f69fe75ad6; no voters:
I20260927 14:56:50.957723 1554 leader_election.cc:288] T 00000000000000000000000000000000 P a4ce5e5dceed4f60bbfaa9f69fe75ad6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:56:50.957826 1559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4ce5e5dceed4f60bbfaa9f69fe75ad6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:56:50.958014 1559 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4ce5e5dceed4f60bbfaa9f69fe75ad6 [term 1 LEADER]: Becoming Leader. State: Replica: a4ce5e5dceed4f60bbfaa9f69fe75ad6, State: Running, Role: LEADER
I20260927 14:56:50.958319 1559 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4ce5e5dceed4f60bbfaa9f69fe75ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ce5e5dceed4f60bbfaa9f69fe75ad6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33685 } }
I20260927 14:56:50.958935 1554 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4ce5e5dceed4f60bbfaa9f69fe75ad6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:56:50.959564 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4ce5e5dceed4f60bbfaa9f69fe75ad6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4ce5e5dceed4f60bbfaa9f69fe75ad6. Latest consensus state: current_term: 1 leader_uuid: "a4ce5e5dceed4f60bbfaa9f69fe75ad6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ce5e5dceed4f60bbfaa9f69fe75ad6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33685 } } }
I20260927 14:56:50.959678 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4ce5e5dceed4f60bbfaa9f69fe75ad6 [sys.catalog]: This master's current role is: LEADER
I20260927 14:56:50.959930 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4ce5e5dceed4f60bbfaa9f69fe75ad6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4ce5e5dceed4f60bbfaa9f69fe75ad6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ce5e5dceed4f60bbfaa9f69fe75ad6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33685 } } }
I20260927 14:56:50.960034 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4ce5e5dceed4f60bbfaa9f69fe75ad6 [sys.catalog]: This master's current role is: LEADER
I20260927 14:56:50.960155 1565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:56:50.963083 1565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:56:50.969035 1565 catalog_manager.cc:1347] Generated new cluster ID: f12bcee5b0094bf99b35c00bd4145743
I20260927 14:56:50.969129 1565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:56:50.982336 1565 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:56:50.983633 1565 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:56:50.990412 1565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4ce5e5dceed4f60bbfaa9f69fe75ad6: Generated new TSK 0
I20260927 14:56:50.991195 1565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:56:51.029840 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33685
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:56:51.150236 1580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:56:51.150542 1580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:56:51.150588 1580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:56:51.154738 1580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:56:51.154850 1580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:56:51.154891 1580 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 14:56:51.154925 1580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:56:51.160494 1580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:56:51.161630 1580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:56:51.162671 1580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:56:51.172467 1588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:51.172467 1585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:51.172472 1586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:51.172662 1580 server_base.cc:1034] running on GCE node
I20260927 14:56:51.173188 1580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:56:51.173892 1580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:56:51.175053 1580 hybrid_clock.cc:648] HybridClock initialized: now 1790521011175026 us; error 42 us; skew 500 ppm
Sep 27 14:56:51 dist-test-slave-pz0h krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:56:51.177874 1580 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:56:51.178834 1580 webserver.cc:460] Webserver started at http://127.0.98.1:42805/ using document root <none> and password file <none>
I20260927 14:56:51.179392 1580 fs_manager.cc:359] Metadata directory not provided
I20260927 14:56:51.179455 1580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:56:51.179682 1580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:56:51.181802 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-0/data/instance:
uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2"
format_stamp: "Formatted at 2026-09-27 14:56:51 on dist-test-slave-pz0h"
server_key: "4b13d4edce3572fe9980269d660af96c"
server_key_iv: "52b9b335c09ac6d2c848947c3dc16cd0"
server_key_version: "encryptionkey@0"
I20260927 14:56:51.182394 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2"
format_stamp: "Formatted at 2026-09-27 14:56:51 on dist-test-slave-pz0h"
server_key: "4b13d4edce3572fe9980269d660af96c"
server_key_iv: "52b9b335c09ac6d2c848947c3dc16cd0"
server_key_version: "encryptionkey@0"
I20260927 14:56:51.186201 1580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 14:56:51.188742 1595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:51.189962 1580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:56:51.190073 1580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2"
format_stamp: "Formatted at 2026-09-27 14:56:51 on dist-test-slave-pz0h"
server_key: "4b13d4edce3572fe9980269d660af96c"
server_key_iv: "52b9b335c09ac6d2c848947c3dc16cd0"
server_key_version: "encryptionkey@0"
I20260927 14:56:51.190171 1580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:56:51.205077 1580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:56:51.208304 1580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:56:51.208491 1580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:56:51.210000 1580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:56:51.210055 1580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:51.210109 1580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:56:51.210137 1580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 14:56:51 dist-test-slave-pz0h krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:56:51.220552 1580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43817
I20260927 14:56:51.220624 1710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43817 every 8 connection(s)
I20260927 14:56:51.221469 1580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:56:51.223798 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:51.210943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55497 (local address 127.0.98.62:33685)
0927 14:56:51.211059 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:51.211062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:51.212302 (+ 1240us) server_negotiation.cc:407] Connection header received
0927 14:56:51.213465 (+ 1163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:51.213469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:51.213535 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:51.213640 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:51.215170 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.215711 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.216960 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.217274 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.217370 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:51.220030 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:51.220034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:51.220036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:51.220064 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:51.221531 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:51.222094 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:51.222098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:51.222100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:51.222219 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:51.222618 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:51.222622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:51.222624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:51.222875 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:51.222968 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:51.223477 (+ 509us) server_negotiation.cc:299] Negotiation successful
0927 14:56:51.223631 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 14:56:51.224587 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:51.211555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33685 (local address 127.0.98.1:55497)
0927 14:56:51.212122 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:51.212193 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:51.213232 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:51.213841 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:51.213851 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:51.214316 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:51.215011 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:51.215028 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.215839 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.215843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:51.216835 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:51.216846 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.217445 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.217449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:51.217649 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:51.218258 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:51.218275 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:51.219833 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:51.221649 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:51.221655 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:51.221669 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:51.221956 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:51.222322 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:51.222325 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:51.222328 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:51.222509 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:51.222975 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:51.222982 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:51.223232 (+ 250us) client_negotiation.cc:769] Sending connection context
0927 14:56:51.223393 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 14:56:51.223742 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":143,"threads_started":1}
I20260927 14:56:51.225818 1711 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33685
I20260927 14:56:51.225965 1711 heartbeater.cc:464] Registering TS with master...
I20260927 14:56:51.226475 1711 heartbeater.cc:511] Master 127.0.98.62:33685 requested a full tablet report, sending...
I20260927 14:56:51.227535 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 1580
I20260927 14:56:51.227654 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:56:51.227932 392 external_mini_cluster.cc:1428] Setting key 6139fec7e41f58d4b3aa0cb74c20d346
I20260927 14:56:51.228139 1508 ts_manager.cc:194] Registered new tserver with Master: d00fd17fc3d04c7d9f3e2d3dfa505bc2 (127.0.98.1:43817)
I20260927 14:56:51.229641 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55497
I20260927 14:56:51.236326 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:51.228513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33551 (local address 127.0.98.62:33685)
0927 14:56:51.228630 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:51.228635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:51.228646 (+ 11us) server_negotiation.cc:407] Connection header received
0927 14:56:51.228682 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:51.228685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:51.228722 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:51.228789 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:51.229499 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.230213 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.231433 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.231755 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.231868 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:51.233177 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:51.233181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:51.233183 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:51.233222 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:51.234546 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:51.235085 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:51.235088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:51.235090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:51.235173 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:51.235497 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:51.235501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:51.235502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:51.235706 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:51.235797 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:51.236086 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 14:56:51.236194 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 14:56:51.236418 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:51.228081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33685 (local address 127.0.98.1:33551)
0927 14:56:51.228442 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:51.228457 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:51.228551 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:51.228825 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:51.228828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:51.229026 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:51.229134 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:51.229140 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.230369 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.230373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:51.231293 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:51.231308 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.231890 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.231894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:51.231989 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:51.232551 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:51.232566 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:51.233021 (+ 455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:51.234683 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:51.234688 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:51.234690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:51.234988 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:51.235279 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:51.235282 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:51.235284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:51.235380 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:51.235809 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:51.235813 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:51.235925 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 14:56:51.236046 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 14:56:51.236210 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":19,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:56:51.253777 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33685
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:56:51.365826 1720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:56:51.366117 1720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:56:51.366163 1720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:56:51.370448 1720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:56:51.370528 1720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:56:51.370569 1720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 14:56:51.370608 1720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:56:51.376078 1720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:56:51.377136 1720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:56:51.378126 1720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:56:51.387537 1728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:51.387598 1720 server_base.cc:1034] running on GCE node
W20260927 14:56:51.387527 1725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:51.387545 1726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:51.388162 1720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:56:51.388804 1720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:56:51.389953 1720 hybrid_clock.cc:648] HybridClock initialized: now 1790521011389918 us; error 53 us; skew 500 ppm
Sep 27 14:56:51 dist-test-slave-pz0h krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:56:51.392581 1720 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:56:51.393519 1720 webserver.cc:460] Webserver started at http://127.0.98.2:34731/ using document root <none> and password file <none>
I20260927 14:56:51.394101 1720 fs_manager.cc:359] Metadata directory not provided
I20260927 14:56:51.394171 1720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:56:51.394372 1720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:56:51.396559 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-1/data/instance:
uuid: "4f8f0e75ba9b4805bee3d665b164726e"
format_stamp: "Formatted at 2026-09-27 14:56:51 on dist-test-slave-pz0h"
server_key: "e2e73364685e848e786905c6d52efb19"
server_key_iv: "cccf0def0030f84afcc967ec04730204"
server_key_version: "encryptionkey@0"
I20260927 14:56:51.397117 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4f8f0e75ba9b4805bee3d665b164726e"
format_stamp: "Formatted at 2026-09-27 14:56:51 on dist-test-slave-pz0h"
server_key: "e2e73364685e848e786905c6d52efb19"
server_key_iv: "cccf0def0030f84afcc967ec04730204"
server_key_version: "encryptionkey@0"
I20260927 14:56:51.401150 1720 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 14:56:51.403842 1735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:51.405110 1720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:56:51.405231 1720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "4f8f0e75ba9b4805bee3d665b164726e"
format_stamp: "Formatted at 2026-09-27 14:56:51 on dist-test-slave-pz0h"
server_key: "e2e73364685e848e786905c6d52efb19"
server_key_iv: "cccf0def0030f84afcc967ec04730204"
server_key_version: "encryptionkey@0"
I20260927 14:56:51.405355 1720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:56:51.433699 1720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:56:51.436817 1720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:56:51.436995 1720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:56:51.438410 1720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:56:51.438458 1720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:51.438512 1720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:56:51.438539 1720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 14:56:51 dist-test-slave-pz0h krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:56:51.448583 1720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33747
I20260927 14:56:51.448721 1850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33747 every 8 connection(s)
I20260927 14:56:51.449626 1720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:56:51.450537 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 1720
I20260927 14:56:51.450650 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:56:51.450963 392 external_mini_cluster.cc:1428] Setting key c8cd194e4274aea452432fecff04d133
I20260927 14:56:51.451546 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:51.439202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54933 (local address 127.0.98.62:33685)
0927 14:56:51.439320 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:51.439324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:51.440377 (+ 1053us) server_negotiation.cc:407] Connection header received
0927 14:56:51.441380 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:51.441385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:51.441445 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:51.441505 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:51.442917 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.443521 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.444655 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.444960 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.445051 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:51.447719 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:51.447723 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:51.447725 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:51.447752 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:51.449200 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:51.449837 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:51.449841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:51.449842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:51.449926 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:51.450335 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:51.450338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:51.450340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:51.450586 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:51.450688 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:51.451261 (+ 573us) server_negotiation.cc:299] Negotiation successful
0927 14:56:51.451394 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 14:56:51.452463 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:51.439730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33685 (local address 127.0.98.2:54933)
0927 14:56:51.440217 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:51.440261 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:51.441127 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:51.441849 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:51.441855 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:51.442193 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:51.442752 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:51.442773 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.443642 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.443646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:51.444532 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:51.444542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.445125 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.445129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:51.445309 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:51.445899 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:51.445915 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:51.447538 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:51.449326 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:51.449332 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:51.449347 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:51.449682 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:51.450039 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:51.450043 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:51.450045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:51.450235 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:51.450687 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:51.450693 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:51.450995 (+ 302us) client_negotiation.cc:769] Sending connection context
0927 14:56:51.451182 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 14:56:51.451504 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":133,"threads_started":1}
I20260927 14:56:51.453615 1851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33685
I20260927 14:56:51.453816 1851 heartbeater.cc:464] Registering TS with master...
I20260927 14:56:51.454434 1851 heartbeater.cc:511] Master 127.0.98.62:33685 requested a full tablet report, sending...
I20260927 14:56:51.455541 1507 ts_manager.cc:194] Registered new tserver with Master: 4f8f0e75ba9b4805bee3d665b164726e (127.0.98.2:33747)
I20260927 14:56:51.456319 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54933
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 14:56:51.462764 1858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:51.455942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33685 (local address 127.0.98.2:56783)
0927 14:56:51.456223 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:51.456240 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:51.456387 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:51.456751 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:51.456754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:51.456974 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:51.457074 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:51.457080 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.457795 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.457798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:51.458361 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:51.458369 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.458778 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.458782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:51.458893 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:51.459353 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:51.459366 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:51.459689 (+ 323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:51.461428 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:51.461431 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:51.461432 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:51.461615 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:51.461825 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:51.461826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:51.461828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:51.461921 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:51.462260 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:51.462262 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:51.462336 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 14:56:51.462462 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 14:56:51.462574 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":103,"threads_started":1}
I20260927 14:56:51.462779 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:51.455807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56783 (local address 127.0.98.62:33685)
0927 14:56:51.455937 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:51.455942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:51.456315 (+ 373us) server_negotiation.cc:407] Connection header received
0927 14:56:51.456500 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:51.456503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:51.456549 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:51.456627 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:51.457182 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.457682 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.458466 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.458684 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.458768 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:51.459857 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:51.459861 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:51.459864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:51.459888 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:51.461336 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:51.461695 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:51.461697 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:51.461698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:51.461765 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:51.461989 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:51.461991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:51.461992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:51.462191 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:51.462263 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:51.462519 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 14:56:51.462631 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:56:51.477819 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33685
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:56:51.589761 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:56:51.590061 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:56:51.590116 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:56:51.594390 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:56:51.594468 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:56:51.594542 1860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 14:56:51.594602 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:56:51.600186 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:56:51.601301 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:56:51.602584 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 14:56:51.613018 1860 server_base.cc:1034] running on GCE node
W20260927 14:56:51.612993 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:51.612993 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:51.613039 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:51.613706 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:56:51.614370 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:56:51.615548 1860 hybrid_clock.cc:648] HybridClock initialized: now 1790521011615527 us; error 35 us; skew 500 ppm
Sep 27 14:56:51 dist-test-slave-pz0h krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:56:51.618366 1860 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:56:51.619366 1860 webserver.cc:460] Webserver started at http://127.0.98.3:38837/ using document root <none> and password file <none>
I20260927 14:56:51.619918 1860 fs_manager.cc:359] Metadata directory not provided
I20260927 14:56:51.619984 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:56:51.620183 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:56:51.622251 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-2/data/instance:
uuid: "1074a8d189474852b22186d980575550"
format_stamp: "Formatted at 2026-09-27 14:56:51 on dist-test-slave-pz0h"
server_key: "7b8f56e933862d1acfc8958436f9ba63"
server_key_iv: "01ace59274e4b175289e19794901d889"
server_key_version: "encryptionkey@0"
I20260927 14:56:51.622825 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1074a8d189474852b22186d980575550"
format_stamp: "Formatted at 2026-09-27 14:56:51 on dist-test-slave-pz0h"
server_key: "7b8f56e933862d1acfc8958436f9ba63"
server_key_iv: "01ace59274e4b175289e19794901d889"
server_key_version: "encryptionkey@0"
I20260927 14:56:51.626928 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 14:56:51.629469 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:51.630731 1860 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:56:51.630903 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "1074a8d189474852b22186d980575550"
format_stamp: "Formatted at 2026-09-27 14:56:51 on dist-test-slave-pz0h"
server_key: "7b8f56e933862d1acfc8958436f9ba63"
server_key_iv: "01ace59274e4b175289e19794901d889"
server_key_version: "encryptionkey@0"
I20260927 14:56:51.631021 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:56:51.664513 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:56:51.667827 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:56:51.668028 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:56:51.669723 1860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:56:51.669785 1860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:51.669842 1860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:56:51.669864 1860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 14:56:51 dist-test-slave-pz0h krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:56:51.680832 1860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46867
I20260927 14:56:51.681036 1990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46867 every 8 connection(s)
I20260927 14:56:51.681806 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:56:51.684017 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:51.670473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57243 (local address 127.0.98.62:33685)
0927 14:56:51.670617 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:51.670622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:51.671667 (+ 1045us) server_negotiation.cc:407] Connection header received
0927 14:56:51.672996 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:51.673001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:51.673068 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:51.673165 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:51.674507 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.675198 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.676514 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.676740 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.676841 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:51.680223 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:51.680226 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:51.680229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:51.680258 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:51.681782 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:51.682448 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:51.682453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:51.682455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:51.682558 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:51.682897 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:51.682901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:51.682903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:51.683110 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:51.683197 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:51.683720 (+ 523us) server_negotiation.cc:299] Negotiation successful
0927 14:56:51.683859 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 14:56:51.684648 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 1860
I20260927 14:56:51.684762 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:56:51.685003 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:51.670953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33685 (local address 127.0.98.3:57243)
0927 14:56:51.671458 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:51.671522 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:51.672652 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:51.673386 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:51.673398 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:51.673780 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:51.674322 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:51.674337 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.675345 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.675348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:51.676379 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:51.676391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.676903 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.676907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:51.677081 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:51.677693 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:51.677710 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:51.680014 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:51.681926 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:51.681932 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:51.681945 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:51.682313 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:51.682639 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:51.682642 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:51.682644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:51.682779 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:51.683217 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:51.683224 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:51.683475 (+ 251us) client_negotiation.cc:769] Sending connection context
0927 14:56:51.683640 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 14:56:51.683970 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":110,"threads_started":1}
I20260927 14:56:51.685048 392 external_mini_cluster.cc:1428] Setting key 51a57cc319ac0730e5e2bfae1cd39049
I20260927 14:56:51.686299 1991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33685
I20260927 14:56:51.686453 1991 heartbeater.cc:464] Registering TS with master...
I20260927 14:56:51.687065 1991 heartbeater.cc:511] Master 127.0.98.62:33685 requested a full tablet report, sending...
I20260927 14:56:51.688158 1507 ts_manager.cc:194] Registered new tserver with Master: 1074a8d189474852b22186d980575550 (127.0.98.3:46867)
I20260927 14:56:51.688931 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57243
I20260927 14:56:51.695328 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:51.688709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43003 (local address 127.0.98.62:33685)
0927 14:56:51.688835 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:51.688839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:51.688993 (+ 154us) server_negotiation.cc:407] Connection header received
0927 14:56:51.689122 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:51.689125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:51.689172 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:51.689252 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:51.689697 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.690222 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.691089 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.691317 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.691406 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:51.692473 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:51.692482 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:51.692484 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:51.692513 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:51.693773 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:51.694154 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:51.694157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:51.694159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:51.694237 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:51.694477 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:51.694480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:51.694482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:51.694664 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:51.694751 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:51.695061 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 14:56:51.695167 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 14:56:51.695343 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:51.688648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33685 (local address 127.0.98.3:43003)
0927 14:56:51.688912 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:51.688925 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:51.689026 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:51.689272 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:51.689275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:51.689480 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:51.689583 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:51.689589 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.690351 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.690354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:51.690982 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:51.690991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.691420 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.691424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:51.691513 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:51.691997 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:51.692007 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:51.692341 (+ 334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:51.693873 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:51.693876 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:51.693879 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:51.694070 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:51.694317 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:51.694320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:51.694323 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:51.694400 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:51.694772 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:51.694775 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:51.694897 (+ 122us) client_negotiation.cc:769] Sending connection context
0927 14:56:51.694993 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 14:56:51.695147 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":100,"threads_started":1}
I20260927 14:56:51.700173 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 14:56:51 dist-test-slave-pz0h krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521011, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 14:56:51 dist-test-slave-pz0h krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521011, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:56:51.725241 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:51.717248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33446 (local address 127.0.98.62:33685)
0927 14:56:51.717368 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:51.717371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:51.717383 (+ 12us) server_negotiation.cc:407] Connection header received
0927 14:56:51.717423 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:51.717426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:51.717464 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:51.717532 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:51.718230 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.718755 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.719620 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.719839 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.719932 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:51.722304 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:51.722307 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:51.722310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:51.722340 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:51.723665 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:51.724062 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:51.724066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:51.724067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:51.724144 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:51.724385 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:51.724388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:51.724390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:51.724574 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:51.724688 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:51.724995 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 14:56:51.725098 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 14:56:51.728368 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:56:51.731436 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:56:51.747305 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:51.740902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43992 (local address 127.0.98.3:46867)
0927 14:56:51.741647 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:51.741664 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:51.741696 (+ 32us) server_negotiation.cc:407] Connection header received
0927 14:56:51.741775 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:51.741780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:51.741999 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:51.742554 (+ 555us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:56:51.742696 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.743880 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.745883 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.746623 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.746783 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:51.746966 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:51.747065 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 14:56:51.747163 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":129,"threads_started":1}
I20260927 14:56:51.747316 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:51.741898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51068 (local address 127.0.98.1:43817)
0927 14:56:51.742343 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:51.742350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:51.742399 (+ 49us) server_negotiation.cc:407] Connection header received
0927 14:56:51.742546 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:51.742550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:51.742713 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:51.742924 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:56:51.743240 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.743974 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.745620 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.746401 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.746517 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:51.746681 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:51.747089 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 14:56:51.747193 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":74,"threads_started":1}
I20260927 14:56:51.747301 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:51.740789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33747 (local address 127.0.0.1:58650)
0927 14:56:51.741805 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:51.741818 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:51.741899 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:51.742726 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:51.742729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:51.742743 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:56:51.742874 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:51.742881 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.744181 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.744185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:51.745750 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:51.745762 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.746565 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.746568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:51.746712 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:51.746831 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 14:56:51.746942 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 14:56:51.747108 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"mutex_wait_us":334,"thread_start_us":69,"threads_started":1}
I20260927 14:56:51.747299 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:51.740756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46867 (local address 127.0.0.1:43992)
0927 14:56:51.741154 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:51.741215 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:51.741359 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:51.742201 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:51.742207 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:51.742233 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:56:51.742405 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:51.742412 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.744004 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.744008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:51.745758 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:51.745764 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.746811 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.746815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:51.746929 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:51.746953 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 14:56:51.747032 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 14:56:51.747109 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":182,"thread_start_us":88,"threads_started":1}
I20260927 14:56:51.748282 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:51.741430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58650 (local address 127.0.98.2:33747)
0927 14:56:51.742298 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:51.742310 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:51.742331 (+ 21us) server_negotiation.cc:407] Connection header received
0927 14:56:51.742400 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:51.742405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:51.742593 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:51.742744 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:56:51.742979 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.744031 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.745905 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.746444 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.747877 (+ 1433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:51.748046 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:51.748090 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 14:56:51.748169 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":775,"thread_start_us":109,"threads_started":1}
I20260927 14:56:51.749145 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:51.741544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43817 (local address 127.0.0.1:51068)
0927 14:56:51.742323 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:51.742336 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:51.742430 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:51.743043 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:51.743045 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:51.743060 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:56:51.743134 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:51.743137 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.744097 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.744101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:51.745500 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:51.745509 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.746533 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.746537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:51.746712 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:51.746844 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 14:56:51.746944 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 14:56:51.749017 (+ 2073us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":2111,"thread_start_us":104,"threads_started":1}
I20260927 14:56:51.750536 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 47f3d3c9063c4761834436ad27b3ba00 (DEFAULT_TABLE table=test-table [id=cc189ad6dcbb4576bf5d6c60710410c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:56:51.750774 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 47f3d3c9063c4761834436ad27b3ba00 (DEFAULT_TABLE table=test-table [id=cc189ad6dcbb4576bf5d6c60710410c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:56:51.751673 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47f3d3c9063c4761834436ad27b3ba00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:51.751706 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47f3d3c9063c4761834436ad27b3ba00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:51.751874 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 47f3d3c9063c4761834436ad27b3ba00 (DEFAULT_TABLE table=test-table [id=cc189ad6dcbb4576bf5d6c60710410c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:56:51.752912 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47f3d3c9063c4761834436ad27b3ba00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:51.759119 2014 tablet_bootstrap.cc:492] T 47f3d3c9063c4761834436ad27b3ba00 P 4f8f0e75ba9b4805bee3d665b164726e: Bootstrap starting.
I20260927 14:56:51.760987 2015 tablet_bootstrap.cc:492] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2: Bootstrap starting.
I20260927 14:56:51.762068 2014 tablet_bootstrap.cc:654] T 47f3d3c9063c4761834436ad27b3ba00 P 4f8f0e75ba9b4805bee3d665b164726e: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:51.762933 2014 log.cc:824] T 47f3d3c9063c4761834436ad27b3ba00 P 4f8f0e75ba9b4805bee3d665b164726e: Log is configured to *not* fsync() on all Append() calls
I20260927 14:56:51.763269 2016 tablet_bootstrap.cc:492] T 47f3d3c9063c4761834436ad27b3ba00 P 1074a8d189474852b22186d980575550: Bootstrap starting.
I20260927 14:56:51.763530 2015 tablet_bootstrap.cc:654] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:51.764387 2015 log.cc:824] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2: Log is configured to *not* fsync() on all Append() calls
I20260927 14:56:51.765347 2014 tablet_bootstrap.cc:492] T 47f3d3c9063c4761834436ad27b3ba00 P 4f8f0e75ba9b4805bee3d665b164726e: No bootstrap required, opened a new log
I20260927 14:56:51.765466 2016 tablet_bootstrap.cc:654] T 47f3d3c9063c4761834436ad27b3ba00 P 1074a8d189474852b22186d980575550: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:51.765556 2014 ts_tablet_manager.cc:1397] T 47f3d3c9063c4761834436ad27b3ba00 P 4f8f0e75ba9b4805bee3d665b164726e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 14:56:51.766173 2016 log.cc:824] T 47f3d3c9063c4761834436ad27b3ba00 P 1074a8d189474852b22186d980575550: Log is configured to *not* fsync() on all Append() calls
I20260927 14:56:51.766371 2015 tablet_bootstrap.cc:492] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2: No bootstrap required, opened a new log
I20260927 14:56:51.766572 2015 ts_tablet_manager.cc:1397] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 14:56:51.768080 2016 tablet_bootstrap.cc:492] T 47f3d3c9063c4761834436ad27b3ba00 P 1074a8d189474852b22186d980575550: No bootstrap required, opened a new log
I20260927 14:56:51.768277 2016 ts_tablet_manager.cc:1397] T 47f3d3c9063c4761834436ad27b3ba00 P 1074a8d189474852b22186d980575550: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 14:56:51.769685 2014 raft_consensus.cc:348] T 47f3d3c9063c4761834436ad27b3ba00 P 4f8f0e75ba9b4805bee3d665b164726e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } }
I20260927 14:56:51.770007 2014 raft_consensus.cc:374] T 47f3d3c9063c4761834436ad27b3ba00 P 4f8f0e75ba9b4805bee3d665b164726e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:51.770066 2014 raft_consensus.cc:729] T 47f3d3c9063c4761834436ad27b3ba00 P 4f8f0e75ba9b4805bee3d665b164726e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8f0e75ba9b4805bee3d665b164726e, State: Initialized, Role: FOLLOWER
I20260927 14:56:51.770581 2014 consensus_queue.cc:260] T 47f3d3c9063c4761834436ad27b3ba00 P 4f8f0e75ba9b4805bee3d665b164726e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } }
I20260927 14:56:51.770921 2015 raft_consensus.cc:348] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } }
I20260927 14:56:51.771200 2015 raft_consensus.cc:374] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:51.771256 2015 raft_consensus.cc:729] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00fd17fc3d04c7d9f3e2d3dfa505bc2, State: Initialized, Role: FOLLOWER
I20260927 14:56:51.771576 1851 heartbeater.cc:503] Master 127.0.98.62:33685 was elected leader, sending a full tablet report...
I20260927 14:56:51.771773 2014 ts_tablet_manager.cc:1428] T 47f3d3c9063c4761834436ad27b3ba00 P 4f8f0e75ba9b4805bee3d665b164726e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 14:56:51.771751 2015 consensus_queue.cc:260] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } }
I20260927 14:56:51.772363 1711 heartbeater.cc:503] Master 127.0.98.62:33685 was elected leader, sending a full tablet report...
I20260927 14:56:51.772723 2015 ts_tablet_manager.cc:1428] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 14:56:51.772776 2016 raft_consensus.cc:348] T 47f3d3c9063c4761834436ad27b3ba00 P 1074a8d189474852b22186d980575550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } }
I20260927 14:56:51.773059 2016 raft_consensus.cc:374] T 47f3d3c9063c4761834436ad27b3ba00 P 1074a8d189474852b22186d980575550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:51.773118 2016 raft_consensus.cc:729] T 47f3d3c9063c4761834436ad27b3ba00 P 1074a8d189474852b22186d980575550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1074a8d189474852b22186d980575550, State: Initialized, Role: FOLLOWER
I20260927 14:56:51.773571 2016 consensus_queue.cc:260] T 47f3d3c9063c4761834436ad27b3ba00 P 1074a8d189474852b22186d980575550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } }
I20260927 14:56:51.774286 1991 heartbeater.cc:503] Master 127.0.98.62:33685 was elected leader, sending a full tablet report...
I20260927 14:56:51.774588 2016 ts_tablet_manager.cc:1428] T 47f3d3c9063c4761834436ad27b3ba00 P 1074a8d189474852b22186d980575550: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 14:56:51.780462 2021 raft_consensus.cc:482] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:56:51.780670 2021 raft_consensus.cc:504] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } }
I20260927 14:56:51.781898 2021 leader_election.cc:288] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1074a8d189474852b22186d980575550 (127.0.98.3:46867), 4f8f0e75ba9b4805bee3d665b164726e (127.0.98.2:33747)
I20260927 14:56:51.786262 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:51.782314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41121 (local address 127.0.98.2:33747)
0927 14:56:51.782442 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:51.782447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:51.782642 (+ 195us) server_negotiation.cc:407] Connection header received
0927 14:56:51.782750 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:51.782752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:51.782829 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:51.782914 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:56:51.783333 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.783938 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.785278 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.785710 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.785822 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:51.785891 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:51.786058 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 14:56:51.786134 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 14:56:51.786296 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:51.782179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33747 (local address 127.0.98.1:41121)
0927 14:56:51.782571 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:51.782582 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:51.782662 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:51.782929 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:51.782932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:51.782955 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:56:51.783164 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:51.783172 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.784080 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.784083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:51.785136 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:51.785148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.785843 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.785846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:51.785945 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:51.785966 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 14:56:51.786031 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 14:56:51.786110 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":21,"spinlock_wait_cycles":4224,"thread_start_us":134,"threads_started":1}
I20260927 14:56:51.786393 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:51.782541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37927 (local address 127.0.98.3:46867)
0927 14:56:51.782657 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:51.782661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:51.782674 (+ 13us) server_negotiation.cc:407] Connection header received
0927 14:56:51.782749 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:51.782752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:51.782850 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:51.782915 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:56:51.783783 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.784349 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.785364 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.785807 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.785900 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:51.785962 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:51.786180 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 14:56:51.786258 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 14:56:51.786607 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:51.782113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46867 (local address 127.0.98.1:37927)
0927 14:56:51.782533 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:51.782548 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:51.782661 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:51.782939 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:51.782941 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:51.782961 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:56:51.783642 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:51.783650 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.784455 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.784459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:51.785220 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:51.785232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.785929 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.785932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:51.786044 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:51.786064 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 14:56:51.786163 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 14:56:51.786236 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":59,"spinlock_wait_cycles":1536,"thread_start_us":136,"threads_started":1}
I20260927 14:56:51.786984 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47f3d3c9063c4761834436ad27b3ba00" candidate_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8f0e75ba9b4805bee3d665b164726e" is_pre_election: true
I20260927 14:56:51.787147 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47f3d3c9063c4761834436ad27b3ba00" candidate_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1074a8d189474852b22186d980575550" is_pre_election: true
I20260927 14:56:51.787323 1805 raft_consensus.cc:2454] T 47f3d3c9063c4761834436ad27b3ba00 P 4f8f0e75ba9b4805bee3d665b164726e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d00fd17fc3d04c7d9f3e2d3dfa505bc2 in term 0.
I20260927 14:56:51.787436 1945 raft_consensus.cc:2454] T 47f3d3c9063c4761834436ad27b3ba00 P 1074a8d189474852b22186d980575550 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d00fd17fc3d04c7d9f3e2d3dfa505bc2 in term 0.
I20260927 14:56:51.787827 1596 leader_election.cc:302] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1074a8d189474852b22186d980575550, 4f8f0e75ba9b4805bee3d665b164726e, d00fd17fc3d04c7d9f3e2d3dfa505bc2; no voters:
I20260927 14:56:51.788085 2021 raft_consensus.cc:2781] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:56:51.788166 2021 raft_consensus.cc:482] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:56:51.788216 2021 raft_consensus.cc:3037] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:51.789273 2021 raft_consensus.cc:504] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } }
I20260927 14:56:51.789659 2021 leader_election.cc:288] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [CANDIDATE]: Term 1 election: Requested vote from peers 1074a8d189474852b22186d980575550 (127.0.98.3:46867), 4f8f0e75ba9b4805bee3d665b164726e (127.0.98.2:33747)
I20260927 14:56:51.790099 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47f3d3c9063c4761834436ad27b3ba00" candidate_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8f0e75ba9b4805bee3d665b164726e"
I20260927 14:56:51.790114 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47f3d3c9063c4761834436ad27b3ba00" candidate_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1074a8d189474852b22186d980575550"
I20260927 14:56:51.790236 1945 raft_consensus.cc:3037] T 47f3d3c9063c4761834436ad27b3ba00 P 1074a8d189474852b22186d980575550 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:51.790246 1805 raft_consensus.cc:3037] T 47f3d3c9063c4761834436ad27b3ba00 P 4f8f0e75ba9b4805bee3d665b164726e [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:51.791621 1805 raft_consensus.cc:2454] T 47f3d3c9063c4761834436ad27b3ba00 P 4f8f0e75ba9b4805bee3d665b164726e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d00fd17fc3d04c7d9f3e2d3dfa505bc2 in term 1.
I20260927 14:56:51.791615 1945 raft_consensus.cc:2454] T 47f3d3c9063c4761834436ad27b3ba00 P 1074a8d189474852b22186d980575550 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d00fd17fc3d04c7d9f3e2d3dfa505bc2 in term 1.
I20260927 14:56:51.791965 1599 leader_election.cc:302] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1074a8d189474852b22186d980575550, d00fd17fc3d04c7d9f3e2d3dfa505bc2; no voters:
I20260927 14:56:51.792198 2021 raft_consensus.cc:2781] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:56:51.792420 2021 raft_consensus.cc:686] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 1 LEADER]: Becoming Leader. State: Replica: d00fd17fc3d04c7d9f3e2d3dfa505bc2, State: Running, Role: LEADER
I20260927 14:56:51.792791 2021 consensus_queue.cc:237] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } }
I20260927 14:56:51.796288 1508 catalog_manager.cc:5495] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d00fd17fc3d04c7d9f3e2d3dfa505bc2 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } health_report { overall_health: HEALTHY } } }
I20260927 14:56:51.818681 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:51.814948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51082 (local address 127.0.98.1:43817)
0927 14:56:51.815078 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:51.815082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:51.815099 (+ 17us) server_negotiation.cc:407] Connection header received
0927 14:56:51.815355 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:51.815358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:51.815415 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:51.815499 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:56:51.815752 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.816289 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.817170 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.817398 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.817485 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:51.817826 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:56:51.818281 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:56:51.818382 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:51.818477 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 14:56:51.818540 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 14:56:51.825078 1945 raft_consensus.cc:1261] T 47f3d3c9063c4761834436ad27b3ba00 P 1074a8d189474852b22186d980575550 [term 1 FOLLOWER]: Refusing update from remote peer d00fd17fc3d04c7d9f3e2d3dfa505bc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:56:51.825090 1805 raft_consensus.cc:1261] T 47f3d3c9063c4761834436ad27b3ba00 P 4f8f0e75ba9b4805bee3d665b164726e [term 1 FOLLOWER]: Refusing update from remote peer d00fd17fc3d04c7d9f3e2d3dfa505bc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:56:51.825781 2025 consensus_queue.cc:1035] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:56:51.826017 2021 consensus_queue.cc:1035] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:56:51.835732 2030 mvcc.cc:205] Tried to move back new op lower bound from 7333974064426819584 to 7333974064306405376. Current Snapshot: MvccSnapshot[applied={T|T < 7333974064426819584}]
I20260927 14:56:51.846949 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:56:51.847183 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:56:51.849905 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4ce5e5dceed4f60bbfaa9f69fe75ad6: Sending DeleteTablet for 3 replicas of tablet 47f3d3c9063c4761834436ad27b3ba00
I20260927 14:56:51.850759 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 47f3d3c9063c4761834436ad27b3ba00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:56:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43992
I20260927 14:56:51.850745 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 47f3d3c9063c4761834436ad27b3ba00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:56:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58650
I20260927 14:56:51.850737 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 47f3d3c9063c4761834436ad27b3ba00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:56:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51068
I20260927 14:56:51.851503 2038 tablet_replica.cc:330] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2: stopping tablet replica
I20260927 14:56:51.851497 2039 tablet_replica.cc:330] T 47f3d3c9063c4761834436ad27b3ba00 P 1074a8d189474852b22186d980575550: stopping tablet replica
I20260927 14:56:51.851871 2039 raft_consensus.cc:2229] T 47f3d3c9063c4761834436ad27b3ba00 P 1074a8d189474852b22186d980575550 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:56:51.851871 2038 raft_consensus.cc:2229] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:56:51.852310 2039 raft_consensus.cc:2258] T 47f3d3c9063c4761834436ad27b3ba00 P 1074a8d189474852b22186d980575550 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:56:51.852258 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 14:56:51.852722 2037 tablet_replica.cc:330] T 47f3d3c9063c4761834436ad27b3ba00 P 4f8f0e75ba9b4805bee3d665b164726e: stopping tablet replica
W20260927 14:56:51.852726 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:56:51.852759 2038 raft_consensus.cc:2258] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:56:51.853053 2037 raft_consensus.cc:2229] T 47f3d3c9063c4761834436ad27b3ba00 P 4f8f0e75ba9b4805bee3d665b164726e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:56:51.853382 2037 raft_consensus.cc:2258] T 47f3d3c9063c4761834436ad27b3ba00 P 4f8f0e75ba9b4805bee3d665b164726e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:56:51.855041 2037 ts_tablet_manager.cc:1905] T 47f3d3c9063c4761834436ad27b3ba00 P 4f8f0e75ba9b4805bee3d665b164726e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:56:51.855042 2039 ts_tablet_manager.cc:1905] T 47f3d3c9063c4761834436ad27b3ba00 P 1074a8d189474852b22186d980575550: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:56:51.857969 2037 ts_tablet_manager.cc:1918] T 47f3d3c9063c4761834436ad27b3ba00 P 4f8f0e75ba9b4805bee3d665b164726e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:56:51.857986 2039 ts_tablet_manager.cc:1918] T 47f3d3c9063c4761834436ad27b3ba00 P 1074a8d189474852b22186d980575550: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:56:51.858076 2037 log.cc:1196] T 47f3d3c9063c4761834436ad27b3ba00 P 4f8f0e75ba9b4805bee3d665b164726e: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-1/wal/wals/47f3d3c9063c4761834436ad27b3ba00
I20260927 14:56:51.858081 2039 log.cc:1196] T 47f3d3c9063c4761834436ad27b3ba00 P 1074a8d189474852b22186d980575550: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-2/wal/wals/47f3d3c9063c4761834436ad27b3ba00
I20260927 14:56:51.858394 2039 ts_tablet_manager.cc:1939] T 47f3d3c9063c4761834436ad27b3ba00 P 1074a8d189474852b22186d980575550: Deleting consensus metadata
I20260927 14:56:51.858415 2037 ts_tablet_manager.cc:1939] T 47f3d3c9063c4761834436ad27b3ba00 P 4f8f0e75ba9b4805bee3d665b164726e: Deleting consensus metadata
I20260927 14:56:51.859917 1492 catalog_manager.cc:4841] TS 4f8f0e75ba9b4805bee3d665b164726e (127.0.98.2:33747): tablet 47f3d3c9063c4761834436ad27b3ba00 (table test-table [id=cc189ad6dcbb4576bf5d6c60710410c1]) successfully deleted
I20260927 14:56:51.860369 2038 ts_tablet_manager.cc:1905] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:56:51.863389 2038 ts_tablet_manager.cc:1918] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:56:51.863515 2038 log.cc:1196] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-0/wal/wals/47f3d3c9063c4761834436ad27b3ba00
I20260927 14:56:51.863802 2038 ts_tablet_manager.cc:1939] T 47f3d3c9063c4761834436ad27b3ba00 P d00fd17fc3d04c7d9f3e2d3dfa505bc2: Deleting consensus metadata
I20260927 14:56:51.864019 1495 catalog_manager.cc:4841] TS 1074a8d189474852b22186d980575550 (127.0.98.3:46867): tablet 47f3d3c9063c4761834436ad27b3ba00 (table test-table [id=cc189ad6dcbb4576bf5d6c60710410c1]) successfully deleted
I20260927 14:56:51.864990 1493 catalog_manager.cc:4841] TS d00fd17fc3d04c7d9f3e2d3dfa505bc2 (127.0.98.1:43817): tablet 47f3d3c9063c4761834436ad27b3ba00 (table test-table [id=cc189ad6dcbb4576bf5d6c60710410c1]) successfully deleted
I20260927 14:56:51.866918 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 91a17e41db4f4192a3b6c86f3ca94f8b (DEFAULT_TABLE table=test-table [id=ee19b318f6c14a97823c6fd8978e3454]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:56:51.866997 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 91a17e41db4f4192a3b6c86f3ca94f8b (DEFAULT_TABLE table=test-table [id=ee19b318f6c14a97823c6fd8978e3454]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:56:51.867210 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a17e41db4f4192a3b6c86f3ca94f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:51.867278 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a17e41db4f4192a3b6c86f3ca94f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:51.869380 2015 tablet_bootstrap.cc:492] T 91a17e41db4f4192a3b6c86f3ca94f8b P d00fd17fc3d04c7d9f3e2d3dfa505bc2: Bootstrap starting.
I20260927 14:56:51.869791 2014 tablet_bootstrap.cc:492] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e: Bootstrap starting.
I20260927 14:56:51.869877 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 91a17e41db4f4192a3b6c86f3ca94f8b (DEFAULT_TABLE table=test-table [id=ee19b318f6c14a97823c6fd8978e3454]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:56:51.870141 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a17e41db4f4192a3b6c86f3ca94f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:51.870457 2015 tablet_bootstrap.cc:654] T 91a17e41db4f4192a3b6c86f3ca94f8b P d00fd17fc3d04c7d9f3e2d3dfa505bc2: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:51.870700 2014 tablet_bootstrap.cc:654] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:51.872605 2016 tablet_bootstrap.cc:492] T 91a17e41db4f4192a3b6c86f3ca94f8b P 1074a8d189474852b22186d980575550: Bootstrap starting.
I20260927 14:56:51.872637 2014 tablet_bootstrap.cc:492] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e: No bootstrap required, opened a new log
I20260927 14:56:51.872704 2015 tablet_bootstrap.cc:492] T 91a17e41db4f4192a3b6c86f3ca94f8b P d00fd17fc3d04c7d9f3e2d3dfa505bc2: No bootstrap required, opened a new log
I20260927 14:56:51.872720 2014 ts_tablet_manager.cc:1397] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 14:56:51.872774 2015 ts_tablet_manager.cc:1397] T 91a17e41db4f4192a3b6c86f3ca94f8b P d00fd17fc3d04c7d9f3e2d3dfa505bc2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 14:56:51.873178 2014 raft_consensus.cc:348] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } }
I20260927 14:56:51.873194 2015 raft_consensus.cc:348] T 91a17e41db4f4192a3b6c86f3ca94f8b P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } }
I20260927 14:56:51.873292 2014 raft_consensus.cc:374] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:51.873292 2015 raft_consensus.cc:374] T 91a17e41db4f4192a3b6c86f3ca94f8b P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:51.873323 2015 raft_consensus.cc:729] T 91a17e41db4f4192a3b6c86f3ca94f8b P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00fd17fc3d04c7d9f3e2d3dfa505bc2, State: Initialized, Role: FOLLOWER
I20260927 14:56:51.873334 2014 raft_consensus.cc:729] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8f0e75ba9b4805bee3d665b164726e, State: Initialized, Role: FOLLOWER
I20260927 14:56:51.873447 2014 consensus_queue.cc:260] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } }
I20260927 14:56:51.873456 2015 consensus_queue.cc:260] T 91a17e41db4f4192a3b6c86f3ca94f8b P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } }
I20260927 14:56:51.873588 2016 tablet_bootstrap.cc:654] T 91a17e41db4f4192a3b6c86f3ca94f8b P 1074a8d189474852b22186d980575550: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:51.873692 2014 ts_tablet_manager.cc:1428] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:56:51.873693 2015 ts_tablet_manager.cc:1428] T 91a17e41db4f4192a3b6c86f3ca94f8b P d00fd17fc3d04c7d9f3e2d3dfa505bc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:56:51.875865 2016 tablet_bootstrap.cc:492] T 91a17e41db4f4192a3b6c86f3ca94f8b P 1074a8d189474852b22186d980575550: No bootstrap required, opened a new log
I20260927 14:56:51.875954 2016 ts_tablet_manager.cc:1397] T 91a17e41db4f4192a3b6c86f3ca94f8b P 1074a8d189474852b22186d980575550: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 14:56:51.876377 2016 raft_consensus.cc:348] T 91a17e41db4f4192a3b6c86f3ca94f8b P 1074a8d189474852b22186d980575550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } }
I20260927 14:56:51.876477 2016 raft_consensus.cc:374] T 91a17e41db4f4192a3b6c86f3ca94f8b P 1074a8d189474852b22186d980575550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:51.876505 2016 raft_consensus.cc:729] T 91a17e41db4f4192a3b6c86f3ca94f8b P 1074a8d189474852b22186d980575550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1074a8d189474852b22186d980575550, State: Initialized, Role: FOLLOWER
I20260927 14:56:51.876623 2016 consensus_queue.cc:260] T 91a17e41db4f4192a3b6c86f3ca94f8b P 1074a8d189474852b22186d980575550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } }
I20260927 14:56:51.876837 2016 ts_tablet_manager.cc:1428] T 91a17e41db4f4192a3b6c86f3ca94f8b P 1074a8d189474852b22186d980575550: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:56:51.886253 2020 raft_consensus.cc:482] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:56:51.886422 2020 raft_consensus.cc:504] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } }
I20260927 14:56:51.887609 2020 leader_election.cc:288] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d00fd17fc3d04c7d9f3e2d3dfa505bc2 (127.0.98.1:43817), 1074a8d189474852b22186d980575550 (127.0.98.3:46867)
I20260927 14:56:51.892127 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:51.887834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43817 (local address 127.0.98.2:41359)
0927 14:56:51.888020 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:51.888036 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:51.888144 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:51.888532 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:51.888535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:51.888549 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:56:51.888645 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:51.888650 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.889484 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.889488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:51.890974 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:51.890987 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.891675 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.891679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:51.891786 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:51.891805 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 14:56:51.891875 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 14:56:51.891966 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":4}
I20260927 14:56:51.892108 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:51.887933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41359 (local address 127.0.98.1:43817)
0927 14:56:51.888149 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:51.888153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:51.888169 (+ 16us) server_negotiation.cc:407] Connection header received
0927 14:56:51.888285 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:51.888289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:51.888337 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:51.888414 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:56:51.888764 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.889351 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.891120 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.891551 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.891652 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:51.891736 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:51.891902 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 14:56:51.891974 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260927 14:56:51.892840 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a17e41db4f4192a3b6c86f3ca94f8b" candidate_uuid: "4f8f0e75ba9b4805bee3d665b164726e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" is_pre_election: true
I20260927 14:56:51.893186 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:51.887880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46867 (local address 127.0.98.2:54515)
0927 14:56:51.888348 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:51.888361 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:51.888449 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:51.888711 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:51.888714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:51.888729 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:56:51.888833 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:51.888838 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.890145 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.890149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:51.891864 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:51.891877 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.892800 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.892802 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:51.892885 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:51.892910 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 14:56:51.892969 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 14:56:51.893037 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":6,"thread_start_us":87,"threads_started":1}
I20260927 14:56:51.893152 1665 raft_consensus.cc:2454] T 91a17e41db4f4192a3b6c86f3ca94f8b P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f8f0e75ba9b4805bee3d665b164726e in term 0.
I20260927 14:56:51.893199 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:51.888281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54515 (local address 127.0.98.3:46867)
0927 14:56:51.888406 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:51.888410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:51.888426 (+ 16us) server_negotiation.cc:407] Connection header received
0927 14:56:51.888555 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:51.888558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:51.888606 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:51.888687 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:56:51.888929 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.889847 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.892034 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.892673 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.892795 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:51.892858 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:51.893001 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 14:56:51.893081 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 14:56:51.893586 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a17e41db4f4192a3b6c86f3ca94f8b" candidate_uuid: "4f8f0e75ba9b4805bee3d665b164726e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1074a8d189474852b22186d980575550" is_pre_election: true
I20260927 14:56:51.893762 1944 raft_consensus.cc:2454] T 91a17e41db4f4192a3b6c86f3ca94f8b P 1074a8d189474852b22186d980575550 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f8f0e75ba9b4805bee3d665b164726e in term 0.
I20260927 14:56:51.893781 1737 leader_election.cc:302] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f8f0e75ba9b4805bee3d665b164726e, d00fd17fc3d04c7d9f3e2d3dfa505bc2; no voters:
I20260927 14:56:51.894232 2020 raft_consensus.cc:2781] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:56:51.894286 2020 raft_consensus.cc:482] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:56:51.894323 2020 raft_consensus.cc:3037] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:51.895422 2020 raft_consensus.cc:504] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } }
I20260927 14:56:51.895782 2020 leader_election.cc:288] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e [CANDIDATE]: Term 1 election: Requested vote from peers d00fd17fc3d04c7d9f3e2d3dfa505bc2 (127.0.98.1:43817), 1074a8d189474852b22186d980575550 (127.0.98.3:46867)
I20260927 14:56:51.896150 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a17e41db4f4192a3b6c86f3ca94f8b" candidate_uuid: "4f8f0e75ba9b4805bee3d665b164726e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1074a8d189474852b22186d980575550"
I20260927 14:56:51.896166 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a17e41db4f4192a3b6c86f3ca94f8b" candidate_uuid: "4f8f0e75ba9b4805bee3d665b164726e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2"
I20260927 14:56:51.896250 1944 raft_consensus.cc:3037] T 91a17e41db4f4192a3b6c86f3ca94f8b P 1074a8d189474852b22186d980575550 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:51.896286 1665 raft_consensus.cc:3037] T 91a17e41db4f4192a3b6c86f3ca94f8b P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:51.897269 1944 raft_consensus.cc:2454] T 91a17e41db4f4192a3b6c86f3ca94f8b P 1074a8d189474852b22186d980575550 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f8f0e75ba9b4805bee3d665b164726e in term 1.
I20260927 14:56:51.897431 1665 raft_consensus.cc:2454] T 91a17e41db4f4192a3b6c86f3ca94f8b P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f8f0e75ba9b4805bee3d665b164726e in term 1.
I20260927 14:56:51.897748 1739 leader_election.cc:302] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1074a8d189474852b22186d980575550, 4f8f0e75ba9b4805bee3d665b164726e; no voters:
I20260927 14:56:51.897907 2020 raft_consensus.cc:2781] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:56:51.898113 2020 raft_consensus.cc:686] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e [term 1 LEADER]: Becoming Leader. State: Replica: 4f8f0e75ba9b4805bee3d665b164726e, State: Running, Role: LEADER
I20260927 14:56:51.898362 2020 consensus_queue.cc:237] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } }
I20260927 14:56:51.900782 1508 catalog_manager.cc:5495] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f8f0e75ba9b4805bee3d665b164726e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4f8f0e75ba9b4805bee3d665b164726e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } health_report { overall_health: UNKNOWN } } }
W20260927 14:56:51.932862 1992 tablet.cc:2369] T 91a17e41db4f4192a3b6c86f3ca94f8b P 1074a8d189474852b22186d980575550: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 14:56:51 dist-test-slave-pz0h krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:56:51.945165 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:51.936647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33685 (local address 127.0.0.1:33452)
0927 14:56:51.937389 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:51.937404 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:51.937531 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:51.937760 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:51.937763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:51.938138 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:51.938238 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:51.938245 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.939037 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.939040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:51.939586 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:51.939594 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.940003 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.940007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:51.940108 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:51.940658 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:51.940671 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:51.942082 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:51.943664 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:51.943674 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:51.943680 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:51.943908 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:51.944154 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:51.944156 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:51.944158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:51.944240 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:51.944587 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:51.944593 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:51.944702 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 14:56:51.944807 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 14:56:51.944973 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":17,"thread_start_us":127,"threads_started":1}
I20260927 14:56:51.945165 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:51.936696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33452 (local address 127.0.98.62:33685)
0927 14:56:51.936812 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:51.936817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:51.937493 (+ 676us) server_negotiation.cc:407] Connection header received
0927 14:56:51.937616 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:51.937620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:51.937659 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:51.937730 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:51.938353 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.938934 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.939690 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:51.939906 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:51.939990 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:51.942234 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:51.942238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:51.942240 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:51.942261 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:51.943562 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:51.943994 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:51.943997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:51.943999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:51.944078 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:51.944312 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:51.944315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:51.944317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:51.944494 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:51.944579 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:51.944864 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 14:56:51.945024 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":44}
I20260927 14:56:51.949196 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33452:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 14:56:51.949970 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260927 14:56:51.950920 1852 tablet.cc:2369] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:56:51.954970 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 0196e0c4ee03483ca07e8874ce16db4e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=317cc894599c4b309d9e463d883bd64b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 14:56:51.955129 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 0196e0c4ee03483ca07e8874ce16db4e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=317cc894599c4b309d9e463d883bd64b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 14:56:51.955166 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 0196e0c4ee03483ca07e8874ce16db4e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=317cc894599c4b309d9e463d883bd64b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 14:56:51.955404 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0196e0c4ee03483ca07e8874ce16db4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:51.955443 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0196e0c4ee03483ca07e8874ce16db4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:51.955539 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0196e0c4ee03483ca07e8874ce16db4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:51.958364 2015 tablet_bootstrap.cc:492] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2: Bootstrap starting.
I20260927 14:56:51.958598 2014 tablet_bootstrap.cc:492] T 0196e0c4ee03483ca07e8874ce16db4e P 4f8f0e75ba9b4805bee3d665b164726e: Bootstrap starting.
I20260927 14:56:51.958648 2016 tablet_bootstrap.cc:492] T 0196e0c4ee03483ca07e8874ce16db4e P 1074a8d189474852b22186d980575550: Bootstrap starting.
I20260927 14:56:51.959638 2016 tablet_bootstrap.cc:654] T 0196e0c4ee03483ca07e8874ce16db4e P 1074a8d189474852b22186d980575550: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:51.959774 2014 tablet_bootstrap.cc:654] T 0196e0c4ee03483ca07e8874ce16db4e P 4f8f0e75ba9b4805bee3d665b164726e: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:51.960282 2015 tablet_bootstrap.cc:654] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:51.961632 2014 tablet_bootstrap.cc:492] T 0196e0c4ee03483ca07e8874ce16db4e P 4f8f0e75ba9b4805bee3d665b164726e: No bootstrap required, opened a new log
I20260927 14:56:51.961727 2014 ts_tablet_manager.cc:1397] T 0196e0c4ee03483ca07e8874ce16db4e P 4f8f0e75ba9b4805bee3d665b164726e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 14:56:51.961930 2016 tablet_bootstrap.cc:492] T 0196e0c4ee03483ca07e8874ce16db4e P 1074a8d189474852b22186d980575550: No bootstrap required, opened a new log
I20260927 14:56:51.962019 2016 ts_tablet_manager.cc:1397] T 0196e0c4ee03483ca07e8874ce16db4e P 1074a8d189474852b22186d980575550: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 14:56:51.962244 2014 raft_consensus.cc:348] T 0196e0c4ee03483ca07e8874ce16db4e P 4f8f0e75ba9b4805bee3d665b164726e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } }
I20260927 14:56:51.962397 2014 raft_consensus.cc:374] T 0196e0c4ee03483ca07e8874ce16db4e P 4f8f0e75ba9b4805bee3d665b164726e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:51.962437 2014 raft_consensus.cc:729] T 0196e0c4ee03483ca07e8874ce16db4e P 4f8f0e75ba9b4805bee3d665b164726e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8f0e75ba9b4805bee3d665b164726e, State: Initialized, Role: FOLLOWER
I20260927 14:56:51.962450 2016 raft_consensus.cc:348] T 0196e0c4ee03483ca07e8874ce16db4e P 1074a8d189474852b22186d980575550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } }
I20260927 14:56:51.962558 2016 raft_consensus.cc:374] T 0196e0c4ee03483ca07e8874ce16db4e P 1074a8d189474852b22186d980575550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:51.962594 2016 raft_consensus.cc:729] T 0196e0c4ee03483ca07e8874ce16db4e P 1074a8d189474852b22186d980575550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1074a8d189474852b22186d980575550, State: Initialized, Role: FOLLOWER
I20260927 14:56:51.962571 2014 consensus_queue.cc:260] T 0196e0c4ee03483ca07e8874ce16db4e P 4f8f0e75ba9b4805bee3d665b164726e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } }
I20260927 14:56:51.962718 2016 consensus_queue.cc:260] T 0196e0c4ee03483ca07e8874ce16db4e P 1074a8d189474852b22186d980575550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } }
I20260927 14:56:51.963030 2016 ts_tablet_manager.cc:1428] T 0196e0c4ee03483ca07e8874ce16db4e P 1074a8d189474852b22186d980575550: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 14:56:51.963135 2015 tablet_bootstrap.cc:492] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2: No bootstrap required, opened a new log
I20260927 14:56:51.963235 2015 ts_tablet_manager.cc:1397] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 14:56:51.963248 2022 tablet_replica.cc:439] T 0196e0c4ee03483ca07e8874ce16db4e P 1074a8d189474852b22186d980575550: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } } }
I20260927 14:56:51.963390 2022 tablet_replica.cc:442] T 0196e0c4ee03483ca07e8874ce16db4e P 1074a8d189474852b22186d980575550: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:56:51.963698 2015 raft_consensus.cc:348] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } }
I20260927 14:56:51.963811 2015 raft_consensus.cc:374] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:51.963842 2015 raft_consensus.cc:729] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00fd17fc3d04c7d9f3e2d3dfa505bc2, State: Initialized, Role: FOLLOWER
I20260927 14:56:51.963953 2015 consensus_queue.cc:260] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } }
I20260927 14:56:51.964226 2015 ts_tablet_manager.cc:1428] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:56:51.964442 2025 tablet_replica.cc:439] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } } }
I20260927 14:56:51.964581 2025 tablet_replica.cc:442] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:56:51.965106 2020 tablet_replica.cc:439] T 0196e0c4ee03483ca07e8874ce16db4e P 4f8f0e75ba9b4805bee3d665b164726e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } } }
I20260927 14:56:51.965273 2020 tablet_replica.cc:442] T 0196e0c4ee03483ca07e8874ce16db4e P 4f8f0e75ba9b4805bee3d665b164726e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:56:51.965857 2014 ts_tablet_manager.cc:1428] T 0196e0c4ee03483ca07e8874ce16db4e P 4f8f0e75ba9b4805bee3d665b164726e: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
W20260927 14:56:51.972929 1712 tablet.cc:2369] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:56:52.191483 2025 raft_consensus.cc:482] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:56:52.191614 2025 raft_consensus.cc:504] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } }
I20260927 14:56:52.192036 2025 leader_election.cc:288] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f8f0e75ba9b4805bee3d665b164726e (127.0.98.2:33747), 1074a8d189474852b22186d980575550 (127.0.98.3:46867)
I20260927 14:56:52.192454 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0196e0c4ee03483ca07e8874ce16db4e" candidate_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8f0e75ba9b4805bee3d665b164726e" is_pre_election: true
I20260927 14:56:52.192461 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0196e0c4ee03483ca07e8874ce16db4e" candidate_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1074a8d189474852b22186d980575550" is_pre_election: true
I20260927 14:56:52.192581 1805 raft_consensus.cc:2454] T 0196e0c4ee03483ca07e8874ce16db4e P 4f8f0e75ba9b4805bee3d665b164726e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d00fd17fc3d04c7d9f3e2d3dfa505bc2 in term 0.
I20260927 14:56:52.192605 1944 raft_consensus.cc:2454] T 0196e0c4ee03483ca07e8874ce16db4e P 1074a8d189474852b22186d980575550 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d00fd17fc3d04c7d9f3e2d3dfa505bc2 in term 0.
I20260927 14:56:52.192868 1599 leader_election.cc:302] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1074a8d189474852b22186d980575550, d00fd17fc3d04c7d9f3e2d3dfa505bc2; no voters:
I20260927 14:56:52.193030 2025 raft_consensus.cc:2781] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:56:52.193100 2025 raft_consensus.cc:482] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:56:52.193143 2025 raft_consensus.cc:3037] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:52.194160 2025 raft_consensus.cc:504] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } }
I20260927 14:56:52.194522 2025 leader_election.cc:288] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [CANDIDATE]: Term 1 election: Requested vote from peers 4f8f0e75ba9b4805bee3d665b164726e (127.0.98.2:33747), 1074a8d189474852b22186d980575550 (127.0.98.3:46867)
I20260927 14:56:52.194871 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0196e0c4ee03483ca07e8874ce16db4e" candidate_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8f0e75ba9b4805bee3d665b164726e"
I20260927 14:56:52.194883 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0196e0c4ee03483ca07e8874ce16db4e" candidate_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1074a8d189474852b22186d980575550"
I20260927 14:56:52.194976 1944 raft_consensus.cc:3037] T 0196e0c4ee03483ca07e8874ce16db4e P 1074a8d189474852b22186d980575550 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:52.194979 1805 raft_consensus.cc:3037] T 0196e0c4ee03483ca07e8874ce16db4e P 4f8f0e75ba9b4805bee3d665b164726e [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:52.196050 1805 raft_consensus.cc:2454] T 0196e0c4ee03483ca07e8874ce16db4e P 4f8f0e75ba9b4805bee3d665b164726e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d00fd17fc3d04c7d9f3e2d3dfa505bc2 in term 1.
I20260927 14:56:52.196060 1944 raft_consensus.cc:2454] T 0196e0c4ee03483ca07e8874ce16db4e P 1074a8d189474852b22186d980575550 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d00fd17fc3d04c7d9f3e2d3dfa505bc2 in term 1.
I20260927 14:56:52.196291 1599 leader_election.cc:302] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1074a8d189474852b22186d980575550, d00fd17fc3d04c7d9f3e2d3dfa505bc2; no voters:
I20260927 14:56:52.196444 2025 raft_consensus.cc:2781] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:56:52.196528 2025 raft_consensus.cc:686] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 1 LEADER]: Becoming Leader. State: Replica: d00fd17fc3d04c7d9f3e2d3dfa505bc2, State: Running, Role: LEADER
I20260927 14:56:52.196660 2025 consensus_queue.cc:237] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } }
I20260927 14:56:52.197202 2021 tablet_replica.cc:439] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2: TxnStatusTablet state changed. Reason: New leader d00fd17fc3d04c7d9f3e2d3dfa505bc2. Latest consensus state: current_term: 1 leader_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } } }
I20260927 14:56:52.197304 2021 tablet_replica.cc:442] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2: This TxnStatusTablet replica's current role is: LEADER
I20260927 14:56:52.197566 2056 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 14:56:52.197751 2056 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 14:56:52.198145 1508 catalog_manager.cc:5495] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d00fd17fc3d04c7d9f3e2d3dfa505bc2 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } health_report { overall_health: UNKNOWN } } }
I20260927 14:56:52.261938 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:52.258592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51090 (local address 127.0.98.1:43817)
0927 14:56:52.258716 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:52.258721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:52.258738 (+ 17us) server_negotiation.cc:407] Connection header received
0927 14:56:52.258974 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:52.258977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:52.259022 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:52.259083 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:56:52.259317 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:52.260035 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:52.260877 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:52.261157 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:52.261239 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:52.261516 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:56:52.261592 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:56:52.261649 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:52.261744 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 14:56:52.261795 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 14:56:52.261950 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:52.258522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43817 (local address 127.0.0.1:51090)
0927 14:56:52.258679 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:52.258696 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:52.258843 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:52.259109 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:52.259112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:52.259120 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 14:56:52.259218 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:52.259223 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:52.260128 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:52.260132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:52.260760 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:52.260769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:52.261259 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:52.261262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:52.261344 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:52.261391 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 14:56:52.261660 (+ 269us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 14:56:52.261676 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 14:56:52.261735 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 14:56:52.261836 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 14:56:52.264755 1805 raft_consensus.cc:1261] T 0196e0c4ee03483ca07e8874ce16db4e P 4f8f0e75ba9b4805bee3d665b164726e [term 1 FOLLOWER]: Refusing update from remote peer d00fd17fc3d04c7d9f3e2d3dfa505bc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:56:52.264824 1944 raft_consensus.cc:1261] T 0196e0c4ee03483ca07e8874ce16db4e P 1074a8d189474852b22186d980575550 [term 1 FOLLOWER]: Refusing update from remote peer d00fd17fc3d04c7d9f3e2d3dfa505bc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:56:52.265174 2025 consensus_queue.cc:1035] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:56:52.265332 2021 consensus_queue.cc:1035] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:56:52.266664 2022 tablet_replica.cc:439] T 0196e0c4ee03483ca07e8874ce16db4e P 1074a8d189474852b22186d980575550: TxnStatusTablet state changed. Reason: New leader d00fd17fc3d04c7d9f3e2d3dfa505bc2. Latest consensus state: current_term: 1 leader_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } } }
I20260927 14:56:52.266784 2022 tablet_replica.cc:442] T 0196e0c4ee03483ca07e8874ce16db4e P 1074a8d189474852b22186d980575550: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:56:52.267580 2020 tablet_replica.cc:439] T 0196e0c4ee03483ca07e8874ce16db4e P 4f8f0e75ba9b4805bee3d665b164726e: TxnStatusTablet state changed. Reason: New leader d00fd17fc3d04c7d9f3e2d3dfa505bc2. Latest consensus state: current_term: 1 leader_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } } }
I20260927 14:56:52.267696 2020 tablet_replica.cc:442] T 0196e0c4ee03483ca07e8874ce16db4e P 4f8f0e75ba9b4805bee3d665b164726e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:56:52.268486 2025 tablet_replica.cc:439] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } } }
I20260927 14:56:52.268606 2025 tablet_replica.cc:442] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2: This TxnStatusTablet replica's current role is: LEADER
I20260927 14:56:52.270161 2022 tablet_replica.cc:439] T 0196e0c4ee03483ca07e8874ce16db4e P 1074a8d189474852b22186d980575550: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } } }
I20260927 14:56:52.270287 2022 tablet_replica.cc:442] T 0196e0c4ee03483ca07e8874ce16db4e P 1074a8d189474852b22186d980575550: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:56:52.270377 2029 mvcc.cc:205] Tried to move back new op lower bound from 7333974066232238080 to 7333974065958682624. Current Snapshot: MvccSnapshot[applied={T|T < 7333974066232238080 or (T in {7333974066232238080})}]
I20260927 14:56:52.270584 2028 mvcc.cc:205] Tried to move back new op lower bound from 7333974066232238080 to 7333974065958682624. Current Snapshot: MvccSnapshot[applied={T|T < 7333974066232238080 or (T in {7333974066232238080})}]
I20260927 14:56:52.270767 2021 tablet_replica.cc:439] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } } }
I20260927 14:56:52.270910 2021 tablet_replica.cc:442] T 0196e0c4ee03483ca07e8874ce16db4e P d00fd17fc3d04c7d9f3e2d3dfa505bc2: This TxnStatusTablet replica's current role is: LEADER
I20260927 14:56:52.272119 2020 tablet_replica.cc:439] T 0196e0c4ee03483ca07e8874ce16db4e P 4f8f0e75ba9b4805bee3d665b164726e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 } } peers { permanent_uuid: "4f8f0e75ba9b4805bee3d665b164726e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33747 } } peers { permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 } } }
I20260927 14:56:52.272248 2020 tablet_replica.cc:442] T 0196e0c4ee03483ca07e8874ce16db4e P 4f8f0e75ba9b4805bee3d665b164726e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:56:52.276371 2020 consensus_queue.cc:1035] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1074a8d189474852b22186d980575550" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:56:52.280834 2044 consensus_queue.cc:1035] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d00fd17fc3d04c7d9f3e2d3dfa505bc2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:56:52.281450 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:52.275082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58662 (local address 127.0.98.2:33747)
0927 14:56:52.275202 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:52.275207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:52.275402 (+ 195us) server_negotiation.cc:407] Connection header received
0927 14:56:52.275484 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:52.275486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:52.275546 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:52.275648 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:56:52.276002 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:52.276678 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:52.277690 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:52.278038 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:52.278137 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:52.278510 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:56:52.279123 (+ 613us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:56:52.279240 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:52.281242 (+ 2002us) server_negotiation.cc:299] Negotiation successful
0927 14:56:52.281317 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 14:56:52.293167 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:52.289325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43817 (local address 127.0.98.2:45801)
0927 14:56:52.289664 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:52.289676 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:52.289765 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:52.290087 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:52.290090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:52.290098 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 14:56:52.290211 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:52.290216 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:52.291142 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:52.291145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:52.291838 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:52.291848 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:52.292414 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:52.292417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:52.292490 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:52.292539 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 14:56:52.292857 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 14:56:52.292864 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 14:56:52.292928 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 14:56:52.293010 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":91,"threads_started":1}
I20260927 14:56:52.293179 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:52.289468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45801 (local address 127.0.98.1:43817)
0927 14:56:52.289579 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:52.289584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:52.289740 (+ 156us) server_negotiation.cc:407] Connection header received
0927 14:56:52.289851 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:52.289854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:52.289909 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:52.289991 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:56:52.290314 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:52.291019 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:52.291952 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:52.292302 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:52.292406 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:52.292667 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:56:52.292759 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:56:52.292828 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:52.292949 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 14:56:52.293013 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 14:56:52.322335 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:52.318344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54173 (local address 127.0.98.2:33747)
0927 14:56:52.318467 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:52.318471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:52.318671 (+ 200us) server_negotiation.cc:407] Connection header received
0927 14:56:52.318852 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:52.318855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:52.318906 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:52.318967 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:56:52.319407 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:52.320040 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:52.321265 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:52.321491 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:52.321572 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:52.321918 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:56:52.321988 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:56:52.322033 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:52.322148 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 14:56:52.322208 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 14:56:52.322410 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:52.318205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33747 (local address 127.0.98.1:54173)
0927 14:56:52.318580 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:52.318598 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:52.318723 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:52.319070 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:52.319073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:52.319082 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 14:56:52.319261 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:52.319268 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:52.320173 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:52.320178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:52.321134 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:52.321147 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:52.321614 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:52.321618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:52.321722 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:52.321791 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 14:56:52.322065 (+ 274us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 14:56:52.322072 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 14:56:52.322142 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 14:56:52.322250 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":129,"threads_started":1}
I20260927 14:56:52.352240 1505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:56:52.352424 1505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:56:52.354965 1505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4ce5e5dceed4f60bbfaa9f69fe75ad6: Sending DeleteTablet for 3 replicas of tablet 91a17e41db4f4192a3b6c86f3ca94f8b
I20260927 14:56:52.355937 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 91a17e41db4f4192a3b6c86f3ca94f8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:56:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43992
I20260927 14:56:52.356144 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 91a17e41db4f4192a3b6c86f3ca94f8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:56:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58650
I20260927 14:56:52.356346 2039 tablet_replica.cc:330] T 91a17e41db4f4192a3b6c86f3ca94f8b P 1074a8d189474852b22186d980575550: stopping tablet replica
I20260927 14:56:52.356472 2039 raft_consensus.cc:2229] T 91a17e41db4f4192a3b6c86f3ca94f8b P 1074a8d189474852b22186d980575550 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:56:52.356606 2039 raft_consensus.cc:2258] T 91a17e41db4f4192a3b6c86f3ca94f8b P 1074a8d189474852b22186d980575550 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:56:52.356774 2037 tablet_replica.cc:330] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e: stopping tablet replica
I20260927 14:56:52.356894 2037 raft_consensus.cc:2229] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:56:52.357215 2037 raft_consensus.cc:2258] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:56:52.358022 2039 ts_tablet_manager.cc:1905] T 91a17e41db4f4192a3b6c86f3ca94f8b P 1074a8d189474852b22186d980575550: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:56:52.358250 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 91a17e41db4f4192a3b6c86f3ca94f8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:56:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51068
I20260927 14:56:52.358263 2037 ts_tablet_manager.cc:1905] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:56:52.358444 2038 tablet_replica.cc:330] T 91a17e41db4f4192a3b6c86f3ca94f8b P d00fd17fc3d04c7d9f3e2d3dfa505bc2: stopping tablet replica
I20260927 14:56:52.358562 2038 raft_consensus.cc:2229] T 91a17e41db4f4192a3b6c86f3ca94f8b P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:56:52.358711 2038 raft_consensus.cc:2258] T 91a17e41db4f4192a3b6c86f3ca94f8b P d00fd17fc3d04c7d9f3e2d3dfa505bc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:56:52.360625 2038 ts_tablet_manager.cc:1905] T 91a17e41db4f4192a3b6c86f3ca94f8b P d00fd17fc3d04c7d9f3e2d3dfa505bc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:56:52.361079 2037 ts_tablet_manager.cc:1918] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 14:56:52.361158 2037 log.cc:1196] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-1/wal/wals/91a17e41db4f4192a3b6c86f3ca94f8b
I20260927 14:56:52.361166 2039 ts_tablet_manager.cc:1918] T 91a17e41db4f4192a3b6c86f3ca94f8b P 1074a8d189474852b22186d980575550: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 14:56:52.361240 2039 log.cc:1196] T 91a17e41db4f4192a3b6c86f3ca94f8b P 1074a8d189474852b22186d980575550: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-2/wal/wals/91a17e41db4f4192a3b6c86f3ca94f8b
I20260927 14:56:52.361390 2037 ts_tablet_manager.cc:1939] T 91a17e41db4f4192a3b6c86f3ca94f8b P 4f8f0e75ba9b4805bee3d665b164726e: Deleting consensus metadata
I20260927 14:56:52.361495 2039 ts_tablet_manager.cc:1939] T 91a17e41db4f4192a3b6c86f3ca94f8b P 1074a8d189474852b22186d980575550: Deleting consensus metadata
I20260927 14:56:52.362124 1492 catalog_manager.cc:4841] TS 4f8f0e75ba9b4805bee3d665b164726e (127.0.98.2:33747): tablet 91a17e41db4f4192a3b6c86f3ca94f8b (table test-table [id=ee19b318f6c14a97823c6fd8978e3454]) successfully deleted
I20260927 14:56:52.362382 1495 catalog_manager.cc:4841] TS 1074a8d189474852b22186d980575550 (127.0.98.3:46867): tablet 91a17e41db4f4192a3b6c86f3ca94f8b (table test-table [id=ee19b318f6c14a97823c6fd8978e3454]) successfully deleted
I20260927 14:56:52.363472 2038 ts_tablet_manager.cc:1918] T 91a17e41db4f4192a3b6c86f3ca94f8b P d00fd17fc3d04c7d9f3e2d3dfa505bc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 14:56:52.363550 2038 log.cc:1196] T 91a17e41db4f4192a3b6c86f3ca94f8b P d00fd17fc3d04c7d9f3e2d3dfa505bc2: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790521004237705-392-0/minicluster-data/ts-0/wal/wals/91a17e41db4f4192a3b6c86f3ca94f8b
I20260927 14:56:52.363785 2038 ts_tablet_manager.cc:1939] T 91a17e41db4f4192a3b6c86f3ca94f8b P d00fd17fc3d04c7d9f3e2d3dfa505bc2: Deleting consensus metadata
I20260927 14:56:52.364487 1493 catalog_manager.cc:4841] TS d00fd17fc3d04c7d9f3e2d3dfa505bc2 (127.0.98.1:43817): tablet 91a17e41db4f4192a3b6c86f3ca94f8b (table test-table [id=ee19b318f6c14a97823c6fd8978e3454]) successfully deleted
Sep 27 14:56:52 dist-test-slave-pz0h krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521011, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 14:56:52.372502 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:52.363042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51098 (local address 127.0.98.1:43817)
0927 14:56:52.363155 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:52.363160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:52.363486 (+ 326us) server_negotiation.cc:407] Connection header received
0927 14:56:52.363650 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:52.363654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:52.363704 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:52.363754 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:52.364643 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:52.365171 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:52.365976 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:52.366182 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:52.366265 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:52.369082 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:52.369091 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:52.369104 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:52.369127 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:52.370866 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:52.371257 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:52.371263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:52.371264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:52.371342 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:52.371580 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:52.371583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:52.371585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:52.371843 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:52.372033 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:52.372219 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 14:56:52.372341 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 14:56:52.372944 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51098
I20260927 14:56:52.382027 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:52.374575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33468 (local address 127.0.98.62:33685)
0927 14:56:52.374696 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:52.374701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:52.374751 (+ 50us) server_negotiation.cc:407] Connection header received
0927 14:56:52.374923 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:52.374926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:52.374979 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:52.375056 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:52.375625 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:52.376441 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:52.377163 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:52.377502 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:52.377600 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:52.378771 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:52.378775 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:52.378778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:52.378867 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:52.380314 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:52.380789 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:52.380793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:52.380795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:52.380892 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:52.381148 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:52.381151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:52.381153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:52.381394 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:52.381528 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:52.381735 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 14:56:52.381859 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 14:56:52.382462 1505 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33468
I20260927 14:56:52.383783 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 1580
I20260927 14:56:52.393604 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 1720
I20260927 14:56:52.398921 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 1860
I20260927 14:56:52.403925 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 1476
2026-09-27T14:56:52Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2936 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:56:52 dist-test-slave-pz0h krb5kdc[2078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:56:52 dist-test-slave-pz0h krb5kdc[2078](info): set up 2 sockets
Sep 27 14:56:52 dist-test-slave-pz0h krb5kdc[2078](info): commencing operation
krb5kdc: starting...
W20260927 14:56:53.362505 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 14:56:53 dist-test-slave-pz0h krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:56:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:56:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:56:53.482214 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40439
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:56:53.596835 2091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:56:53.597137 2091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:56:53.597188 2091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:56:53.601442 2091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:56:53.601527 2091 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 14:56:53.601544 2091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:56:53.601562 2091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:56:53.601576 2091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:56:53.601590 2091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:56:53.607283 2091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40439
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:56:53.608485 2091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:56:53.609606 2091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 14:56:53.618681 2091 server_base.cc:1034] running on GCE node
W20260927 14:56:53.618595 2097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:53.618700 2099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:53.618593 2096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:53.619575 2091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:56:53.620842 2091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:56:53.622044 2091 hybrid_clock.cc:648] HybridClock initialized: now 1790521013622010 us; error 53 us; skew 500 ppm
Sep 27 14:56:53 dist-test-slave-pz0h krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:56:53.625110 2091 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:56:53.626148 2091 webserver.cc:460] Webserver started at http://127.0.98.62:33279/ using document root <none> and password file <none>
I20260927 14:56:53.626757 2091 fs_manager.cc:359] Metadata directory not provided
I20260927 14:56:53.626876 2091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:56:53.627076 2091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:56:53.629232 2091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "132a9d51505746b4a773d526a0d7c474"
format_stamp: "Formatted at 2026-09-27 14:56:53 on dist-test-slave-pz0h"
server_key: "72c1f3becd9d50da014291ca6ed187f5"
server_key_iv: "3725b6ae6042d0f1080e77b8edc0a50c"
server_key_version: "encryptionkey@0"
I20260927 14:56:53.629782 2091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "132a9d51505746b4a773d526a0d7c474"
format_stamp: "Formatted at 2026-09-27 14:56:53 on dist-test-slave-pz0h"
server_key: "72c1f3becd9d50da014291ca6ed187f5"
server_key_iv: "3725b6ae6042d0f1080e77b8edc0a50c"
server_key_version: "encryptionkey@0"
I20260927 14:56:53.633773 2091 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 14:56:53.636478 2106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:53.637842 2091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 14:56:53.637982 2091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "132a9d51505746b4a773d526a0d7c474"
format_stamp: "Formatted at 2026-09-27 14:56:53 on dist-test-slave-pz0h"
server_key: "72c1f3becd9d50da014291ca6ed187f5"
server_key_iv: "3725b6ae6042d0f1080e77b8edc0a50c"
server_key_version: "encryptionkey@0"
I20260927 14:56:53.638105 2091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:56:53.649844 2091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:56:53.653358 2091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:56:53.653585 2091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:56:53.662035 2091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40439
I20260927 14:56:53.662045 2168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40439 every 8 connection(s)
I20260927 14:56:53.663157 2091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:56:53.666034 2169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:53.668885 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 2091
I20260927 14:56:53.669023 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 14:56:53.669291 392 external_mini_cluster.cc:1428] Setting key 58ebd994e7b77af02b68bbe044fbaddf
I20260927 14:56:53.673557 2169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 132a9d51505746b4a773d526a0d7c474: Bootstrap starting.
I20260927 14:56:53.676494 2169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 132a9d51505746b4a773d526a0d7c474: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:53.677363 2169 log.cc:824] T 00000000000000000000000000000000 P 132a9d51505746b4a773d526a0d7c474: Log is configured to *not* fsync() on all Append() calls
Sep 27 14:56:53 dist-test-slave-pz0h krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:56:53.679842 2169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 132a9d51505746b4a773d526a0d7c474: No bootstrap required, opened a new log
I20260927 14:56:53.684286 2169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 132a9d51505746b4a773d526a0d7c474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "132a9d51505746b4a773d526a0d7c474" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40439 } }
I20260927 14:56:53.684753 2169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 132a9d51505746b4a773d526a0d7c474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:53.684842 2169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 132a9d51505746b4a773d526a0d7c474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 132a9d51505746b4a773d526a0d7c474, State: Initialized, Role: FOLLOWER
I20260927 14:56:53.685432 2169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 132a9d51505746b4a773d526a0d7c474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "132a9d51505746b4a773d526a0d7c474" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40439 } }
I20260927 14:56:53.685631 2169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 132a9d51505746b4a773d526a0d7c474 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:56:53.685745 2169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 132a9d51505746b4a773d526a0d7c474 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:56:53.685879 2169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 132a9d51505746b4a773d526a0d7c474 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:53.687136 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:53.670755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50352 (local address 127.0.98.62:40439)
0927 14:56:53.671255 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:53.671273 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:53.671312 (+ 39us) server_negotiation.cc:407] Connection header received
0927 14:56:53.671998 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:53.672018 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:53.672440 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:53.672779 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:53.673789 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:53.674871 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:53.676099 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:53.676696 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:53.676878 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:53.679479 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:53.679486 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:53.679504 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:53.679541 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:53.682538 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:53.683538 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:53.683557 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:53.683636 (+ 79us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:53.683905 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:53.684401 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:53.684406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:53.684407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:53.685056 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:53.685302 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:53.685625 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 14:56:53.685987 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":134,"threads_started":1}
I20260927 14:56:53.687203 2169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 132a9d51505746b4a773d526a0d7c474 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "132a9d51505746b4a773d526a0d7c474" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40439 } }
I20260927 14:56:53.687599 2169 leader_election.cc:302] T 00000000000000000000000000000000 P 132a9d51505746b4a773d526a0d7c474 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 132a9d51505746b4a773d526a0d7c474; no voters:
I20260927 14:56:53.687964 2169 leader_election.cc:288] T 00000000000000000000000000000000 P 132a9d51505746b4a773d526a0d7c474 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:56:53.688087 2174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 132a9d51505746b4a773d526a0d7c474 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:56:53.688321 2174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 132a9d51505746b4a773d526a0d7c474 [term 1 LEADER]: Becoming Leader. State: Replica: 132a9d51505746b4a773d526a0d7c474, State: Running, Role: LEADER
I20260927 14:56:53.688696 2174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 132a9d51505746b4a773d526a0d7c474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "132a9d51505746b4a773d526a0d7c474" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40439 } }
I20260927 14:56:53.689254 2169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 132a9d51505746b4a773d526a0d7c474 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:56:53.690030 2176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 132a9d51505746b4a773d526a0d7c474 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 132a9d51505746b4a773d526a0d7c474. Latest consensus state: current_term: 1 leader_uuid: "132a9d51505746b4a773d526a0d7c474" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "132a9d51505746b4a773d526a0d7c474" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40439 } } }
I20260927 14:56:53.690289 2176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 132a9d51505746b4a773d526a0d7c474 [sys.catalog]: This master's current role is: LEADER
I20260927 14:56:53.690412 2175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 132a9d51505746b4a773d526a0d7c474 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "132a9d51505746b4a773d526a0d7c474" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "132a9d51505746b4a773d526a0d7c474" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40439 } } }
I20260927 14:56:53.690579 2175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 132a9d51505746b4a773d526a0d7c474 [sys.catalog]: This master's current role is: LEADER
I20260927 14:56:53.691090 2183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:56:53.694058 2183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:56:53.700783 2183 catalog_manager.cc:1347] Generated new cluster ID: e695472e7d58443db4a5dbd2e28da006
I20260927 14:56:53.700876 2183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:56:53.730283 2183 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:56:53.731666 2183 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:56:53.739126 2183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 132a9d51505746b4a773d526a0d7c474: Generated new TSK 0
I20260927 14:56:53.739907 2183 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:56:53.774215 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40439
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:56:53.916661 2195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:56:53.916986 2195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:56:53.917037 2195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:56:53.921826 2195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:56:53.921929 2195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:56:53.921980 2195 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 14:56:53.921994 2195 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 14:56:53.922030 2195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:56:53.928668 2195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:56:53.929795 2195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:56:53.931046 2195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:56:53.941146 2200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:53.941193 2201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:53.941144 2203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:53.941212 2195 server_base.cc:1034] running on GCE node
I20260927 14:56:53.941919 2195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:56:53.942653 2195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:56:53.943805 2195 hybrid_clock.cc:648] HybridClock initialized: now 1790521013943788 us; error 36 us; skew 500 ppm
Sep 27 14:56:53 dist-test-slave-pz0h krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:56:53.946640 2195 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:56:53.947620 2195 webserver.cc:460] Webserver started at http://127.0.98.1:37575/ using document root <none> and password file <none>
I20260927 14:56:53.948166 2195 fs_manager.cc:359] Metadata directory not provided
I20260927 14:56:53.948225 2195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:56:53.948388 2195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:56:53.950392 2195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-0/data/instance:
uuid: "44d7da27cda843f7bc138c74ef574202"
format_stamp: "Formatted at 2026-09-27 14:56:53 on dist-test-slave-pz0h"
server_key: "1d3ade4b05c9dd728a5e86e341392d07"
server_key_iv: "c420efd798ebf4a493ce311a5ca0660a"
server_key_version: "encryptionkey@0"
I20260927 14:56:53.950953 2195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "44d7da27cda843f7bc138c74ef574202"
format_stamp: "Formatted at 2026-09-27 14:56:53 on dist-test-slave-pz0h"
server_key: "1d3ade4b05c9dd728a5e86e341392d07"
server_key_iv: "c420efd798ebf4a493ce311a5ca0660a"
server_key_version: "encryptionkey@0"
I20260927 14:56:53.954717 2195 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 14:56:53.957289 2210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:53.958555 2195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 14:56:53.958675 2195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "44d7da27cda843f7bc138c74ef574202"
format_stamp: "Formatted at 2026-09-27 14:56:53 on dist-test-slave-pz0h"
server_key: "1d3ade4b05c9dd728a5e86e341392d07"
server_key_iv: "c420efd798ebf4a493ce311a5ca0660a"
server_key_version: "encryptionkey@0"
I20260927 14:56:53.958775 2195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:56:53.969488 2195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:56:53.972882 2195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:56:53.973047 2195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:56:53.974491 2195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:56:53.974548 2195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:53.974592 2195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:56:53.974622 2195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 14:56:53 dist-test-slave-pz0h krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:56:53.984493 2195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42795
I20260927 14:56:53.984547 2325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42795 every 8 connection(s)
I20260927 14:56:53.985595 2195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:56:53.988188 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:53.975313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47611 (local address 127.0.98.62:40439)
0927 14:56:53.975477 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:53.975482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:53.976390 (+ 908us) server_negotiation.cc:407] Connection header received
0927 14:56:53.977527 (+ 1137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:53.977532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:53.977597 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:53.977696 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:53.979270 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:53.980103 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:53.981386 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:53.981675 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:53.981766 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:53.984596 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:53.984601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:53.984604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:53.984637 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:53.986130 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:53.986689 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:53.986692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:53.986694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:53.986779 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:53.987098 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:53.987100 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:53.987101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:53.987281 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:53.987368 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:53.987941 (+ 573us) server_negotiation.cc:299] Negotiation successful
0927 14:56:53.988060 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 14:56:53.989089 2219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:53.975739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.1:47611)
0927 14:56:53.976201 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:53.976259 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:53.977240 (+ 981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:53.977929 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:53.977939 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:53.978375 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:53.979106 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:53.979127 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:53.980247 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:53.980250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:53.981254 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:53.981265 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:53.981837 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:53.981842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:53.982028 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:53.982715 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:53.982736 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:53.984388 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:53.986273 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:53.986278 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:53.986293 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:53.986593 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:53.986886 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:53.986888 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:53.986890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:53.987027 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:53.987405 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:53.987411 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:53.987666 (+ 255us) client_negotiation.cc:769] Sending connection context
0927 14:56:53.987848 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 14:56:53.988133 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":117,"threads_started":1}
I20260927 14:56:53.990448 2326 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40439
I20260927 14:56:53.990644 2326 heartbeater.cc:464] Registering TS with master...
I20260927 14:56:53.991186 2326 heartbeater.cc:511] Master 127.0.98.62:40439 requested a full tablet report, sending...
I20260927 14:56:53.991912 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 2195
I20260927 14:56:53.992022 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:56:53.992316 392 external_mini_cluster.cc:1428] Setting key 3710f4612fe3f758a074acc96b13072d
I20260927 14:56:53.993317 2122 ts_manager.cc:194] Registered new tserver with Master: 44d7da27cda843f7bc138c74ef574202 (127.0.98.1:42795)
I20260927 14:56:53.994760 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47611
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 14:56:54.002676 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:53.992900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48421 (local address 127.0.98.62:40439)
0927 14:56:53.992999 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:53.993004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:53.993194 (+ 190us) server_negotiation.cc:407] Connection header received
0927 14:56:53.993298 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:53.993301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:53.993344 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:53.993419 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:53.993849 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:53.994354 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:53.995272 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:53.995600 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:53.995668 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:53.999493 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:53.999498 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:53.999501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:53.999532 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:54.000995 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:54.001477 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:54.001480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:54.001482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:54.001560 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:54.001891 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:54.001894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:54.001895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:54.002123 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:54.002229 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:54.002445 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 14:56:54.002548 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":29}
I20260927 14:56:54.002686 2332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:53.992665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.1:48421)
0927 14:56:53.993106 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:53.993120 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:53.993224 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:53.993438 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:53.993441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:53.993640 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:53.993738 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:53.993744 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:53.994468 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:53.994471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:53.995135 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:53.995147 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:53.995825 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:53.995828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:53.995919 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:53.998928 (+ 3009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:53.998941 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:53.999292 (+ 351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:54.001133 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:54.001136 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:54.001138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:54.001376 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:54.001666 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:54.001669 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:54.001671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:54.001765 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:54.002219 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:54.002221 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:54.002309 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 14:56:54.002397 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 14:56:54.002526 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:56:54.017890 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40439
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:56:54.128326 2335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:56:54.128609 2335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:56:54.128660 2335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:56:54.132768 2335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:56:54.132840 2335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:56:54.132881 2335 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 14:56:54.132896 2335 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 14:56:54.132946 2335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:56:54.138860 2335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:56:54.139920 2335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:56:54.141088 2335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:56:54.150765 2340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:54.150766 2343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:54.150774 2341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:54.150836 2335 server_base.cc:1034] running on GCE node
I20260927 14:56:54.151371 2335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:56:54.152009 2335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:56:54.153183 2335 hybrid_clock.cc:648] HybridClock initialized: now 1790521014153164 us; error 32 us; skew 500 ppm
Sep 27 14:56:54 dist-test-slave-pz0h krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:56:54.155952 2335 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:56:54.156839 2335 webserver.cc:460] Webserver started at http://127.0.98.2:44063/ using document root <none> and password file <none>
I20260927 14:56:54.157384 2335 fs_manager.cc:359] Metadata directory not provided
I20260927 14:56:54.157445 2335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:56:54.157648 2335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:56:54.159834 2335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-1/data/instance:
uuid: "c5506b5c58aa4191be250c176296a85f"
format_stamp: "Formatted at 2026-09-27 14:56:54 on dist-test-slave-pz0h"
server_key: "f602343946b483989a6e6862cb71ce90"
server_key_iv: "e84dbc93aeca30997983a526f6c0979b"
server_key_version: "encryptionkey@0"
I20260927 14:56:54.160415 2335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c5506b5c58aa4191be250c176296a85f"
format_stamp: "Formatted at 2026-09-27 14:56:54 on dist-test-slave-pz0h"
server_key: "f602343946b483989a6e6862cb71ce90"
server_key_iv: "e84dbc93aeca30997983a526f6c0979b"
server_key_version: "encryptionkey@0"
I20260927 14:56:54.164381 2335 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 14:56:54.167035 2350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:54.168269 2335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:56:54.168375 2335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "c5506b5c58aa4191be250c176296a85f"
format_stamp: "Formatted at 2026-09-27 14:56:54 on dist-test-slave-pz0h"
server_key: "f602343946b483989a6e6862cb71ce90"
server_key_iv: "e84dbc93aeca30997983a526f6c0979b"
server_key_version: "encryptionkey@0"
I20260927 14:56:54.168494 2335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:56:54.183004 2335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:56:54.186018 2335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:56:54.186223 2335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:56:54.187726 2335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:56:54.187779 2335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:54.187841 2335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:56:54.187865 2335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 14:56:54 dist-test-slave-pz0h krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:56:54.198055 2335 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35587
I20260927 14:56:54.198078 2465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35587 every 8 connection(s)
I20260927 14:56:54.198985 2335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:56:54.201391 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.188479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35161 (local address 127.0.98.62:40439)
0927 14:56:54.188594 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:54.188597 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:54.189700 (+ 1103us) server_negotiation.cc:407] Connection header received
0927 14:56:54.191173 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:54.191177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:54.191242 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:54.191356 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:54.192771 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.193333 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.194538 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.194759 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.194901 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.197806 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:54.197811 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:54.197813 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:54.197841 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:54.199350 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:54.199900 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:54.199903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:54.199905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:54.199993 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:54.200297 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:54.200301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:54.200303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:54.200501 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:54.200585 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:54.201079 (+ 494us) server_negotiation.cc:299] Negotiation successful
0927 14:56:54.201217 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 14:56:54.202154 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.189037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.2:35161)
0927 14:56:54.189537 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:54.189581 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:54.190888 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:54.191559 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:54.191569 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:54.192008 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:54.192614 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:54.192631 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.193450 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.193453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:54.194418 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:54.194428 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.194960 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.194964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:54.195123 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.196194 (+ 1071us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:54.196210 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:54.197626 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:54.199487 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:54.199493 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:54.199506 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:54.199803 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:54.200075 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:54.200078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:54.200080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:54.200212 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:54.200595 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:54.200602 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:54.200840 (+ 238us) client_negotiation.cc:769] Sending connection context
0927 14:56:54.201020 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 14:56:54.201345 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":148,"threads_started":1}
I20260927 14:56:54.203379 2466 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40439
I20260927 14:56:54.203521 2466 heartbeater.cc:464] Registering TS with master...
I20260927 14:56:54.204001 2466 heartbeater.cc:511] Master 127.0.98.62:40439 requested a full tablet report, sending...
I20260927 14:56:54.204245 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 2335
I20260927 14:56:54.204339 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:56:54.204612 392 external_mini_cluster.cc:1428] Setting key dc281e136c9ea9b2b0444248e15be4ba
I20260927 14:56:54.205138 2123 ts_manager.cc:194] Registered new tserver with Master: c5506b5c58aa4191be250c176296a85f (127.0.98.2:35587)
I20260927 14:56:54.205855 2123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35161
I20260927 14:56:54.212921 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.205534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39913 (local address 127.0.98.62:40439)
0927 14:56:54.205739 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:54.205743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:54.205822 (+ 79us) server_negotiation.cc:407] Connection header received
0927 14:56:54.205978 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:54.205982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:54.206025 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:54.206097 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:54.206662 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.207429 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.208361 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.208621 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.208702 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.209925 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:54.209928 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:54.209930 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:54.209951 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:54.211274 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:54.211730 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:54.211733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:54.211734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:54.211799 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:54.212053 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:54.212055 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:54.212056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:54.212270 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:54.212341 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:54.212679 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 14:56:54.212796 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260927 14:56:54.212975 2472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.205470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.2:39913)
0927 14:56:54.205741 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:54.205755 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:54.205888 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:54.206207 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:54.206210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:54.206424 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:54.206543 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:54.206550 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.207577 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.207580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:54.208236 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:54.208245 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.208742 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.208746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:54.208832 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.209316 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:54.209329 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:54.209800 (+ 471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:54.211382 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:54.211385 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:54.211388 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:54.211638 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:54.211873 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:54.211876 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:54.211878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:54.211970 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:54.212357 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:54.212360 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:54.212470 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 14:56:54.212585 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 14:56:54.212761 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:56:54.229260 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40439
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:56:54.339769 2475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:56:54.340061 2475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:56:54.340117 2475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:56:54.344448 2475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:56:54.344521 2475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:56:54.344563 2475 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 14:56:54.344576 2475 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 14:56:54.344631 2475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:56:54.350298 2475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:56:54.351472 2475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:56:54.352530 2475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:56:54.362737 2480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:54.362735 2481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:54.362787 2483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:54.363046 2475 server_base.cc:1034] running on GCE node
I20260927 14:56:54.363600 2475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:56:54.364255 2475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:56:54.365410 2475 hybrid_clock.cc:648] HybridClock initialized: now 1790521014365376 us; error 48 us; skew 500 ppm
Sep 27 14:56:54 dist-test-slave-pz0h krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:56:54.368122 2475 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:56:54.369007 2475 webserver.cc:460] Webserver started at http://127.0.98.3:43025/ using document root <none> and password file <none>
I20260927 14:56:54.369546 2475 fs_manager.cc:359] Metadata directory not provided
I20260927 14:56:54.369609 2475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:56:54.369823 2475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:56:54.372536 2475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-2/data/instance:
uuid: "d19f76cccc1c4fe192f0503263707fae"
format_stamp: "Formatted at 2026-09-27 14:56:54 on dist-test-slave-pz0h"
server_key: "ddda11418cfc8c761dcf2b1fd43c8203"
server_key_iv: "da555c09baf6c1e217389948bd212b47"
server_key_version: "encryptionkey@0"
I20260927 14:56:54.373237 2475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d19f76cccc1c4fe192f0503263707fae"
format_stamp: "Formatted at 2026-09-27 14:56:54 on dist-test-slave-pz0h"
server_key: "ddda11418cfc8c761dcf2b1fd43c8203"
server_key_iv: "da555c09baf6c1e217389948bd212b47"
server_key_version: "encryptionkey@0"
I20260927 14:56:54.377331 2475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 14:56:54.380033 2490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:54.381317 2475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:56:54.381459 2475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "d19f76cccc1c4fe192f0503263707fae"
format_stamp: "Formatted at 2026-09-27 14:56:54 on dist-test-slave-pz0h"
server_key: "ddda11418cfc8c761dcf2b1fd43c8203"
server_key_iv: "da555c09baf6c1e217389948bd212b47"
server_key_version: "encryptionkey@0"
I20260927 14:56:54.381574 2475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:56:54.396313 2475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:56:54.399503 2475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:56:54.399694 2475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:56:54.401127 2475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:56:54.401181 2475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:54.401232 2475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:56:54.401261 2475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 14:56:54 dist-test-slave-pz0h krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:56:54.411235 2475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43043
I20260927 14:56:54.411300 2605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43043 every 8 connection(s)
I20260927 14:56:54.412169 2475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:56:54.414855 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.401890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32901 (local address 127.0.98.62:40439)
0927 14:56:54.402044 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:54.402050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:54.402916 (+ 866us) server_negotiation.cc:407] Connection header received
0927 14:56:54.403966 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:54.403970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:54.404032 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:54.404092 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:54.405806 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.406555 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.407868 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.408096 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.408202 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.411207 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:54.411212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:54.411215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:54.411250 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:54.412744 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:54.413388 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:54.413391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:54.413393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:54.413481 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:54.413776 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:54.413778 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:54.413779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:54.413965 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:54.414038 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:54.414548 (+ 510us) server_negotiation.cc:299] Negotiation successful
0927 14:56:54.414676 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 14:56:54.415541 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 2475
I20260927 14:56:54.415658 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:56:54.415772 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.402280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.3:32901)
0927 14:56:54.402736 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:54.402787 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:54.403715 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:54.404529 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:54.404539 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:54.404956 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:54.405643 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:54.405666 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.406672 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.406684 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:54.407750 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:54.407761 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.408245 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.408250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:54.408450 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.409082 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:54.409105 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:54.410985 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:54.412898 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:54.412904 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:54.412919 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:54.413253 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:54.413558 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:54.413559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:54.413561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:54.413702 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:54.414036 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:54.414041 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:54.414286 (+ 245us) client_negotiation.cc:769] Sending connection context
0927 14:56:54.414437 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 14:56:54.414721 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":115,"threads_started":1}
I20260927 14:56:54.415966 392 external_mini_cluster.cc:1428] Setting key f7f03b6ba6d6a65c37e50135fe16a829
I20260927 14:56:54.417102 2606 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40439
I20260927 14:56:54.417268 2606 heartbeater.cc:464] Registering TS with master...
I20260927 14:56:54.417824 2606 heartbeater.cc:511] Master 127.0.98.62:40439 requested a full tablet report, sending...
I20260927 14:56:54.419023 2121 ts_manager.cc:194] Registered new tserver with Master: d19f76cccc1c4fe192f0503263707fae (127.0.98.3:43043)
I20260927 14:56:54.420315 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:32901
I20260927 14:56:54.426926 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.419400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.98.3:40785)
0927 14:56:54.419722 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:54.419738 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:54.419844 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:54.420097 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:54.420100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:54.420330 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:54.420550 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:54.420560 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.421482 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.421514 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:54.422246 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:54.422256 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.422770 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.422773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:54.422913 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.423424 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:54.423435 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:54.423796 (+ 361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:54.425394 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:54.425397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:54.425399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:54.425597 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:54.425856 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:54.425859 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:54.425860 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:54.425950 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:54.426380 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:54.426383 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:54.426472 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 14:56:54.426571 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 14:56:54.426699 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":98,"threads_started":1}
I20260927 14:56:54.426944 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.419473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40785 (local address 127.0.98.62:40439)
0927 14:56:54.419589 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:54.419593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:54.419795 (+ 202us) server_negotiation.cc:407] Connection header received
0927 14:56:54.419939 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:54.419943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:54.419986 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:54.420066 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:54.420669 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.421331 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.422365 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.422634 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.422720 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.423933 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:54.423937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:54.423939 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:54.423963 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:54.425277 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:54.425677 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:54.425679 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:54.425680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:54.425750 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:54.426027 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:54.426029 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:54.426030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:54.426218 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:54.426301 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:54.426664 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 14:56:54.426776 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 14:56:54.429668 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 14:56:54 dist-test-slave-pz0h krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 14:56:54 dist-test-slave-pz0h krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:56:54.454146 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.446274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50366 (local address 127.0.98.62:40439)
0927 14:56:54.446404 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:54.446408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:54.446419 (+ 11us) server_negotiation.cc:407] Connection header received
0927 14:56:54.446457 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:54.446459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:54.446495 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:54.446556 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:54.447334 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.447862 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.448694 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.448893 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.448956 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.451337 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:54.451340 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:54.451342 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:54.451368 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:54.452627 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:54.453006 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:54.453008 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:54.453009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:54.453079 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:54.453272 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:54.453274 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:54.453276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:54.453450 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:54.453547 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:54.453860 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 14:56:54.453971 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 14:56:54.457170 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:56:54.460273 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:56:54.474524 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.469272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35587 (local address 127.0.0.1:33084)
0927 14:56:54.469522 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:54.469535 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:54.469617 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:54.470550 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:54.470552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:54.470570 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:56:54.470652 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:54.470658 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.471645 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.471648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:54.473172 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:54.473183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.473982 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.473985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:54.474094 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.474133 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 14:56:54.474231 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 14:56:54.474353 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":57,"thread_start_us":70,"threads_started":1}
I20260927 14:56:54.474514 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.469705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33084 (local address 127.0.98.2:35587)
0927 14:56:54.470146 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:54.470157 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:54.470177 (+ 20us) server_negotiation.cc:407] Connection header received
0927 14:56:54.470244 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:54.470249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:54.470416 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:54.470551 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:56:54.470751 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.471541 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.473308 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.473871 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.473969 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.474135 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:54.474276 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 14:56:54.474371 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":107,"threads_started":1}
I20260927 14:56:54.474578 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.468563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42795 (local address 127.0.0.1:32920)
0927 14:56:54.468916 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:54.468967 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:54.469091 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:54.470286 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:54.470292 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:54.470313 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:56:54.470448 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:54.470455 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.471907 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.471911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:54.473212 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:54.473219 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.474184 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.474188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:54.474278 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.474301 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 14:56:54.474350 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 14:56:54.474420 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":115,"thread_start_us":93,"threads_started":1}
I20260927 14:56:54.474905 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.468899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43043 (local address 127.0.0.1:44202)
0927 14:56:54.469269 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:54.469281 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:54.469354 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:54.470433 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:54.470436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:54.470450 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:56:54.470518 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:54.470523 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.471934 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.471938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:54.473157 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:54.473168 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.474378 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.474382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:54.474455 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.474474 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 14:56:54.474532 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 14:56:54.474764 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":96,"thread_start_us":96,"threads_started":1}
I20260927 14:56:54.475376 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.469160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44202 (local address 127.0.98.3:43043)
0927 14:56:54.469829 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:54.469845 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:54.469869 (+ 24us) server_negotiation.cc:407] Connection header received
0927 14:56:54.469953 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:54.469959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:54.470160 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:54.470299 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:56:54.470630 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.471820 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.473258 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.474052 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.474156 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.474325 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:54.475129 (+ 804us) server_negotiation.cc:299] Negotiation successful
0927 14:56:54.475238 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":107,"threads_started":1}
I20260927 14:56:54.475452 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.468733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32920 (local address 127.0.98.1:42795)
0927 14:56:54.469856 (+ 1123us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:54.469871 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:54.469891 (+ 20us) server_negotiation.cc:407] Connection header received
0927 14:56:54.469964 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:54.469969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:54.470142 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:54.470285 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:56:54.470569 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.471801 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.473307 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.474052 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.475019 (+ 967us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.475179 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:54.475221 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 14:56:54.475326 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1027,"thread_start_us":72,"threads_started":1}
I20260927 14:56:54.477487 2400 tablet_service.cc:1467] Processing CreateTablet for tablet df3c7a778ab048188bfd51b51d0537ea (DEFAULT_TABLE table=test-table [id=cc47fe58064c4c1d8a97cddad0d0f1fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:56:54.477807 2260 tablet_service.cc:1467] Processing CreateTablet for tablet df3c7a778ab048188bfd51b51d0537ea (DEFAULT_TABLE table=test-table [id=cc47fe58064c4c1d8a97cddad0d0f1fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:56:54.478637 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df3c7a778ab048188bfd51b51d0537ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:54.478652 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df3c7a778ab048188bfd51b51d0537ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:54.478621 2527 tablet_service.cc:1467] Processing CreateTablet for tablet df3c7a778ab048188bfd51b51d0537ea (DEFAULT_TABLE table=test-table [id=cc47fe58064c4c1d8a97cddad0d0f1fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:56:54.479880 2527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df3c7a778ab048188bfd51b51d0537ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:54.485932 2629 tablet_bootstrap.cc:492] T df3c7a778ab048188bfd51b51d0537ea P 44d7da27cda843f7bc138c74ef574202: Bootstrap starting.
I20260927 14:56:54.488099 2631 tablet_bootstrap.cc:492] T df3c7a778ab048188bfd51b51d0537ea P d19f76cccc1c4fe192f0503263707fae: Bootstrap starting.
I20260927 14:56:54.488099 2630 tablet_bootstrap.cc:492] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f: Bootstrap starting.
I20260927 14:56:54.488299 2629 tablet_bootstrap.cc:654] T df3c7a778ab048188bfd51b51d0537ea P 44d7da27cda843f7bc138c74ef574202: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:54.489089 2629 log.cc:824] T df3c7a778ab048188bfd51b51d0537ea P 44d7da27cda843f7bc138c74ef574202: Log is configured to *not* fsync() on all Append() calls
I20260927 14:56:54.490442 2630 tablet_bootstrap.cc:654] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:54.490446 2631 tablet_bootstrap.cc:654] T df3c7a778ab048188bfd51b51d0537ea P d19f76cccc1c4fe192f0503263707fae: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:54.491350 2631 log.cc:824] T df3c7a778ab048188bfd51b51d0537ea P d19f76cccc1c4fe192f0503263707fae: Log is configured to *not* fsync() on all Append() calls
I20260927 14:56:54.491350 2630 log.cc:824] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f: Log is configured to *not* fsync() on all Append() calls
I20260927 14:56:54.491495 2629 tablet_bootstrap.cc:492] T df3c7a778ab048188bfd51b51d0537ea P 44d7da27cda843f7bc138c74ef574202: No bootstrap required, opened a new log
I20260927 14:56:54.491691 2629 ts_tablet_manager.cc:1397] T df3c7a778ab048188bfd51b51d0537ea P 44d7da27cda843f7bc138c74ef574202: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 14:56:54.493381 2630 tablet_bootstrap.cc:492] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f: No bootstrap required, opened a new log
I20260927 14:56:54.493573 2630 ts_tablet_manager.cc:1397] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 14:56:54.493705 2631 tablet_bootstrap.cc:492] T df3c7a778ab048188bfd51b51d0537ea P d19f76cccc1c4fe192f0503263707fae: No bootstrap required, opened a new log
I20260927 14:56:54.493891 2631 ts_tablet_manager.cc:1397] T df3c7a778ab048188bfd51b51d0537ea P d19f76cccc1c4fe192f0503263707fae: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 14:56:54.494918 2629 raft_consensus.cc:348] T df3c7a778ab048188bfd51b51d0537ea P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:54.495147 2629 raft_consensus.cc:374] T df3c7a778ab048188bfd51b51d0537ea P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:54.495195 2629 raft_consensus.cc:729] T df3c7a778ab048188bfd51b51d0537ea P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d7da27cda843f7bc138c74ef574202, State: Initialized, Role: FOLLOWER
I20260927 14:56:54.495615 2629 consensus_queue.cc:260] T df3c7a778ab048188bfd51b51d0537ea P 44d7da27cda843f7bc138c74ef574202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:54.496232 2326 heartbeater.cc:503] Master 127.0.98.62:40439 was elected leader, sending a full tablet report...
I20260927 14:56:54.496562 2629 ts_tablet_manager.cc:1428] T df3c7a778ab048188bfd51b51d0537ea P 44d7da27cda843f7bc138c74ef574202: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 14:56:54.497558 2631 raft_consensus.cc:348] T df3c7a778ab048188bfd51b51d0537ea P d19f76cccc1c4fe192f0503263707fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:54.497848 2631 raft_consensus.cc:374] T df3c7a778ab048188bfd51b51d0537ea P d19f76cccc1c4fe192f0503263707fae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:54.497910 2631 raft_consensus.cc:729] T df3c7a778ab048188bfd51b51d0537ea P d19f76cccc1c4fe192f0503263707fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19f76cccc1c4fe192f0503263707fae, State: Initialized, Role: FOLLOWER
I20260927 14:56:54.497787 2630 raft_consensus.cc:348] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:54.498056 2630 raft_consensus.cc:374] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:54.498112 2630 raft_consensus.cc:729] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5506b5c58aa4191be250c176296a85f, State: Initialized, Role: FOLLOWER
I20260927 14:56:54.498339 2631 consensus_queue.cc:260] T df3c7a778ab048188bfd51b51d0537ea P d19f76cccc1c4fe192f0503263707fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:54.498517 2630 consensus_queue.cc:260] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:54.499152 2466 heartbeater.cc:503] Master 127.0.98.62:40439 was elected leader, sending a full tablet report...
I20260927 14:56:54.499430 2631 ts_tablet_manager.cc:1428] T df3c7a778ab048188bfd51b51d0537ea P d19f76cccc1c4fe192f0503263707fae: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 14:56:54.499471 2630 ts_tablet_manager.cc:1428] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260927 14:56:54.499732 2606 heartbeater.cc:503] Master 127.0.98.62:40439 was elected leader, sending a full tablet report...
I20260927 14:56:54.634397 2637 raft_consensus.cc:482] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:56:54.634595 2637 raft_consensus.cc:504] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:54.635732 2637 leader_election.cc:288] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d7da27cda843f7bc138c74ef574202 (127.0.98.1:42795), d19f76cccc1c4fe192f0503263707fae (127.0.98.3:43043)
I20260927 14:56:54.639420 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.636091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51193 (local address 127.0.98.1:42795)
0927 14:56:54.636256 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:54.636260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:54.636476 (+ 216us) server_negotiation.cc:407] Connection header received
0927 14:56:54.636591 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:54.636594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:54.636639 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:54.636698 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:56:54.636953 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.637446 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.638496 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.638922 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.639012 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.639074 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:54.639233 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 14:56:54.639311 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 14:56:54.639442 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.636041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42795 (local address 127.0.98.2:51193)
0927 14:56:54.636387 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:54.636397 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:54.636495 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:54.636722 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:54.636724 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:54.636739 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:56:54.636854 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:54.636859 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.637548 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.637551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:54.638373 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:54.638383 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.639027 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.639031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:54.639130 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.639148 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 14:56:54.639211 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 14:56:54.639281 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":29,"thread_start_us":99,"threads_started":1}
I20260927 14:56:54.639580 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.636149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46887 (local address 127.0.98.3:43043)
0927 14:56:54.636255 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:54.636260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:54.636304 (+ 44us) server_negotiation.cc:407] Connection header received
0927 14:56:54.636467 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:54.636470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:54.636511 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:54.636572 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:56:54.636814 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.637322 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.638636 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.639095 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.639190 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.639247 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:54.639407 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 14:56:54.639473 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 14:56:54.639596 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.636038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43043 (local address 127.0.98.2:46887)
0927 14:56:54.636239 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:54.636254 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:54.636358 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:54.636589 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:54.636591 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:54.636607 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:56:54.636712 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:54.636718 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.637423 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.637426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:54.638510 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:54.638519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.639208 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.639211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:54.639305 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.639321 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 14:56:54.639385 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 14:56:54.639455 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260927 14:56:54.640002 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df3c7a778ab048188bfd51b51d0537ea" candidate_uuid: "c5506b5c58aa4191be250c176296a85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d7da27cda843f7bc138c74ef574202" is_pre_election: true
I20260927 14:56:54.640235 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df3c7a778ab048188bfd51b51d0537ea" candidate_uuid: "c5506b5c58aa4191be250c176296a85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f76cccc1c4fe192f0503263707fae" is_pre_election: true
I20260927 14:56:54.640334 2280 raft_consensus.cc:2454] T df3c7a778ab048188bfd51b51d0537ea P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5506b5c58aa4191be250c176296a85f in term 0.
I20260927 14:56:54.640480 2560 raft_consensus.cc:2454] T df3c7a778ab048188bfd51b51d0537ea P d19f76cccc1c4fe192f0503263707fae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5506b5c58aa4191be250c176296a85f in term 0.
I20260927 14:56:54.640829 2353 leader_election.cc:302] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d7da27cda843f7bc138c74ef574202, c5506b5c58aa4191be250c176296a85f; no voters:
I20260927 14:56:54.641124 2637 raft_consensus.cc:2781] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:56:54.641197 2637 raft_consensus.cc:482] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:56:54.641227 2637 raft_consensus.cc:3037] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:54.642287 2637 raft_consensus.cc:504] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:54.642706 2637 leader_election.cc:288] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f [CANDIDATE]: Term 1 election: Requested vote from peers 44d7da27cda843f7bc138c74ef574202 (127.0.98.1:42795), d19f76cccc1c4fe192f0503263707fae (127.0.98.3:43043)
I20260927 14:56:54.643049 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df3c7a778ab048188bfd51b51d0537ea" candidate_uuid: "c5506b5c58aa4191be250c176296a85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f76cccc1c4fe192f0503263707fae"
I20260927 14:56:54.643078 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df3c7a778ab048188bfd51b51d0537ea" candidate_uuid: "c5506b5c58aa4191be250c176296a85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d7da27cda843f7bc138c74ef574202"
I20260927 14:56:54.643162 2560 raft_consensus.cc:3037] T df3c7a778ab048188bfd51b51d0537ea P d19f76cccc1c4fe192f0503263707fae [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:54.643217 2280 raft_consensus.cc:3037] T df3c7a778ab048188bfd51b51d0537ea P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:54.644160 2560 raft_consensus.cc:2454] T df3c7a778ab048188bfd51b51d0537ea P d19f76cccc1c4fe192f0503263707fae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5506b5c58aa4191be250c176296a85f in term 1.
I20260927 14:56:54.644172 2280 raft_consensus.cc:2454] T df3c7a778ab048188bfd51b51d0537ea P 44d7da27cda843f7bc138c74ef574202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5506b5c58aa4191be250c176296a85f in term 1.
I20260927 14:56:54.644474 2353 leader_election.cc:302] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d7da27cda843f7bc138c74ef574202, c5506b5c58aa4191be250c176296a85f; no voters:
I20260927 14:56:54.644672 2637 raft_consensus.cc:2781] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:56:54.644871 2637 raft_consensus.cc:686] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f [term 1 LEADER]: Becoming Leader. State: Replica: c5506b5c58aa4191be250c176296a85f, State: Running, Role: LEADER
I20260927 14:56:54.645234 2637 consensus_queue.cc:237] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:54.648303 2122 catalog_manager.cc:5495] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f reported cstate change: term changed from 0 to 1, leader changed from <none> to c5506b5c58aa4191be250c176296a85f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c5506b5c58aa4191be250c176296a85f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } health_report { overall_health: UNKNOWN } } }
W20260927 14:56:54.663385 2607 tablet.cc:2369] T df3c7a778ab048188bfd51b51d0537ea P d19f76cccc1c4fe192f0503263707fae: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 14:56:54 dist-test-slave-pz0h krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:56:54.692384 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.684459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40439 (local address 127.0.0.1:50376)
0927 14:56:54.684748 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:54.684761 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:54.684892 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:54.685106 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:54.685108 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:54.685485 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:56:54.685577 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:54.685583 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.686307 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.686309 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:54.686991 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:54.687000 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.687385 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.687387 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:54.687463 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.688011 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:56:54.688023 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:56:54.689459 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:56:54.691044 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:54.691052 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:54.691058 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:54.691307 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:54.691513 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:56:54.691515 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:56:54.691516 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:56:54.691585 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:56:54.691887 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:56:54.691893 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:56:54.691996 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 14:56:54.692079 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 14:56:54.692205 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":3,"thread_start_us":120,"threads_started":1}
I20260927 14:56:54.692392 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.684518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50376 (local address 127.0.98.62:40439)
0927 14:56:54.684677 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:54.684682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:54.684837 (+ 155us) server_negotiation.cc:407] Connection header received
0927 14:56:54.684969 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:54.684972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:54.685014 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:54.685081 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:54.685672 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.686226 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.687095 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.687302 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.687374 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.689583 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:54.689587 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:54.689589 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:54.689610 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:54.690964 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:54.691380 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:54.691383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:54.691384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:54.691450 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:54.691640 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:54.691642 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:54.691643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:54.691813 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:54.691879 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:54.692117 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 14:56:54.692254 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":69}
I20260927 14:56:54.696041 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50376:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 14:56:54.696741 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260927 14:56:54.700076 2467 tablet.cc:2369] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:56:54.701207 2527 tablet_service.cc:1467] Processing CreateTablet for tablet 7341dacdc8df4c579080d1e6842dd266 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c1361037d3f74c62b25a178a9dddc082]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 14:56:54.701193 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 7341dacdc8df4c579080d1e6842dd266 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c1361037d3f74c62b25a178a9dddc082]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 14:56:54.701193 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 7341dacdc8df4c579080d1e6842dd266 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c1361037d3f74c62b25a178a9dddc082]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 14:56:54.701875 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7341dacdc8df4c579080d1e6842dd266. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:54.701880 2527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7341dacdc8df4c579080d1e6842dd266. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:54.701881 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7341dacdc8df4c579080d1e6842dd266. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:54.704439 2631 tablet_bootstrap.cc:492] T 7341dacdc8df4c579080d1e6842dd266 P d19f76cccc1c4fe192f0503263707fae: Bootstrap starting.
I20260927 14:56:54.704638 2629 tablet_bootstrap.cc:492] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202: Bootstrap starting.
I20260927 14:56:54.705338 2631 tablet_bootstrap.cc:654] T 7341dacdc8df4c579080d1e6842dd266 P d19f76cccc1c4fe192f0503263707fae: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:54.705411 2629 tablet_bootstrap.cc:654] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:54.706521 2631 tablet_bootstrap.cc:492] T 7341dacdc8df4c579080d1e6842dd266 P d19f76cccc1c4fe192f0503263707fae: No bootstrap required, opened a new log
I20260927 14:56:54.706606 2631 ts_tablet_manager.cc:1397] T 7341dacdc8df4c579080d1e6842dd266 P d19f76cccc1c4fe192f0503263707fae: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 14:56:54.706669 2629 tablet_bootstrap.cc:492] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202: No bootstrap required, opened a new log
I20260927 14:56:54.706753 2629 ts_tablet_manager.cc:1397] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 14:56:54.707037 2631 raft_consensus.cc:348] T 7341dacdc8df4c579080d1e6842dd266 P d19f76cccc1c4fe192f0503263707fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:54.707132 2631 raft_consensus.cc:374] T 7341dacdc8df4c579080d1e6842dd266 P d19f76cccc1c4fe192f0503263707fae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:54.707161 2631 raft_consensus.cc:729] T 7341dacdc8df4c579080d1e6842dd266 P d19f76cccc1c4fe192f0503263707fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19f76cccc1c4fe192f0503263707fae, State: Initialized, Role: FOLLOWER
I20260927 14:56:54.707135 2629 raft_consensus.cc:348] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:54.707228 2629 raft_consensus.cc:374] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:54.707268 2629 raft_consensus.cc:729] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d7da27cda843f7bc138c74ef574202, State: Initialized, Role: FOLLOWER
I20260927 14:56:54.707293 2631 consensus_queue.cc:260] T 7341dacdc8df4c579080d1e6842dd266 P d19f76cccc1c4fe192f0503263707fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:54.707399 2629 consensus_queue.cc:260] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:54.707549 2631 ts_tablet_manager.cc:1428] T 7341dacdc8df4c579080d1e6842dd266 P d19f76cccc1c4fe192f0503263707fae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:56:54.707549 2636 tablet_replica.cc:439] T 7341dacdc8df4c579080d1e6842dd266 P d19f76cccc1c4fe192f0503263707fae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } } }
I20260927 14:56:54.707651 2629 ts_tablet_manager.cc:1428] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:56:54.707712 2636 tablet_replica.cc:442] T 7341dacdc8df4c579080d1e6842dd266 P d19f76cccc1c4fe192f0503263707fae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:56:54.707860 2635 tablet_replica.cc:439] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } } }
I20260927 14:56:54.707983 2635 tablet_replica.cc:442] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:56:54.708221 2630 tablet_bootstrap.cc:492] T 7341dacdc8df4c579080d1e6842dd266 P c5506b5c58aa4191be250c176296a85f: Bootstrap starting.
I20260927 14:56:54.709160 2630 tablet_bootstrap.cc:654] T 7341dacdc8df4c579080d1e6842dd266 P c5506b5c58aa4191be250c176296a85f: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:54.710407 2630 tablet_bootstrap.cc:492] T 7341dacdc8df4c579080d1e6842dd266 P c5506b5c58aa4191be250c176296a85f: No bootstrap required, opened a new log
I20260927 14:56:54.710503 2630 ts_tablet_manager.cc:1397] T 7341dacdc8df4c579080d1e6842dd266 P c5506b5c58aa4191be250c176296a85f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 14:56:54.710992 2630 raft_consensus.cc:348] T 7341dacdc8df4c579080d1e6842dd266 P c5506b5c58aa4191be250c176296a85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:54.711122 2630 raft_consensus.cc:374] T 7341dacdc8df4c579080d1e6842dd266 P c5506b5c58aa4191be250c176296a85f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:54.711160 2630 raft_consensus.cc:729] T 7341dacdc8df4c579080d1e6842dd266 P c5506b5c58aa4191be250c176296a85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5506b5c58aa4191be250c176296a85f, State: Initialized, Role: FOLLOWER
I20260927 14:56:54.711287 2630 consensus_queue.cc:260] T 7341dacdc8df4c579080d1e6842dd266 P c5506b5c58aa4191be250c176296a85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:54.711567 2630 ts_tablet_manager.cc:1428] T 7341dacdc8df4c579080d1e6842dd266 P c5506b5c58aa4191be250c176296a85f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:56:54.711544 2637 tablet_replica.cc:439] T 7341dacdc8df4c579080d1e6842dd266 P c5506b5c58aa4191be250c176296a85f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } } }
I20260927 14:56:54.711665 2637 tablet_replica.cc:442] T 7341dacdc8df4c579080d1e6842dd266 P c5506b5c58aa4191be250c176296a85f: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 14:56:54.736893 2327 tablet.cc:2369] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:56:54.751833 2635 raft_consensus.cc:482] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:56:54.751986 2635 raft_consensus.cc:504] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:54.753049 2635 leader_election.cc:288] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5506b5c58aa4191be250c176296a85f (127.0.98.2:35587), d19f76cccc1c4fe192f0503263707fae (127.0.98.3:43043)
I20260927 14:56:54.757071 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.753392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50079 (local address 127.0.98.3:43043)
0927 14:56:54.753521 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:54.753527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:54.753734 (+ 207us) server_negotiation.cc:407] Connection header received
0927 14:56:54.753927 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:54.753931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:54.753973 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:54.754038 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:56:54.754356 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.755068 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.756159 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.756550 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.756641 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.756704 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:54.756890 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 14:56:54.756955 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 14:56:54.757098 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.753316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43043 (local address 127.0.98.1:50079)
0927 14:56:54.753647 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:54.753663 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:54.753783 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:54.754046 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:54.754049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:54.754060 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:56:54.754244 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:54.754252 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.755187 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.755190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:54.756028 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:54.756039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.756688 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.756692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:54.756781 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.756799 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 14:56:54.756866 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 14:56:54.756937 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":41,"spinlock_wait_cycles":1408,"thread_start_us":86,"threads_started":1}
I20260927 14:56:54.757279 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.753404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34129 (local address 127.0.98.2:35587)
0927 14:56:54.753522 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:54.753527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:54.753784 (+ 257us) server_negotiation.cc:407] Connection header received
0927 14:56:54.753878 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:54.753881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:54.753936 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:54.754022 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:56:54.754750 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.755415 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.756384 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.756781 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.756870 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.756926 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:54.757083 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 14:56:54.757150 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 14:56:54.757427 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.753320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35587 (local address 127.0.98.1:34129)
0927 14:56:54.753712 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:54.753723 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:54.753793 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:54.754033 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:54.754036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:54.754055 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:56:54.754628 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:54.754639 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.755526 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.755530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:54.756263 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:54.756272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.756890 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.756893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:54.756982 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.757000 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 14:56:54.757062 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 14:56:54.757244 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":49,"thread_start_us":74,"threads_started":1}
I20260927 14:56:54.757551 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7341dacdc8df4c579080d1e6842dd266" candidate_uuid: "44d7da27cda843f7bc138c74ef574202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f76cccc1c4fe192f0503263707fae" is_pre_election: true
I20260927 14:56:54.757720 2560 raft_consensus.cc:2454] T 7341dacdc8df4c579080d1e6842dd266 P d19f76cccc1c4fe192f0503263707fae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d7da27cda843f7bc138c74ef574202 in term 0.
I20260927 14:56:54.758011 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7341dacdc8df4c579080d1e6842dd266" candidate_uuid: "44d7da27cda843f7bc138c74ef574202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5506b5c58aa4191be250c176296a85f" is_pre_election: true
I20260927 14:56:54.758097 2214 leader_election.cc:302] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d7da27cda843f7bc138c74ef574202, d19f76cccc1c4fe192f0503263707fae; no voters:
I20260927 14:56:54.758236 2420 raft_consensus.cc:2454] T 7341dacdc8df4c579080d1e6842dd266 P c5506b5c58aa4191be250c176296a85f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d7da27cda843f7bc138c74ef574202 in term 0.
I20260927 14:56:54.758404 2635 raft_consensus.cc:2781] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:56:54.758467 2635 raft_consensus.cc:482] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:56:54.758505 2635 raft_consensus.cc:3037] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:54.759619 2635 raft_consensus.cc:504] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:54.759948 2635 leader_election.cc:288] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202 [CANDIDATE]: Term 1 election: Requested vote from peers c5506b5c58aa4191be250c176296a85f (127.0.98.2:35587), d19f76cccc1c4fe192f0503263707fae (127.0.98.3:43043)
I20260927 14:56:54.760327 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7341dacdc8df4c579080d1e6842dd266" candidate_uuid: "44d7da27cda843f7bc138c74ef574202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5506b5c58aa4191be250c176296a85f"
I20260927 14:56:54.760313 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7341dacdc8df4c579080d1e6842dd266" candidate_uuid: "44d7da27cda843f7bc138c74ef574202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f76cccc1c4fe192f0503263707fae"
I20260927 14:56:54.760444 2420 raft_consensus.cc:3037] T 7341dacdc8df4c579080d1e6842dd266 P c5506b5c58aa4191be250c176296a85f [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:54.760444 2560 raft_consensus.cc:3037] T 7341dacdc8df4c579080d1e6842dd266 P d19f76cccc1c4fe192f0503263707fae [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:54.761546 2560 raft_consensus.cc:2454] T 7341dacdc8df4c579080d1e6842dd266 P d19f76cccc1c4fe192f0503263707fae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d7da27cda843f7bc138c74ef574202 in term 1.
I20260927 14:56:54.761546 2420 raft_consensus.cc:2454] T 7341dacdc8df4c579080d1e6842dd266 P c5506b5c58aa4191be250c176296a85f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d7da27cda843f7bc138c74ef574202 in term 1.
I20260927 14:56:54.761860 2213 leader_election.cc:302] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d7da27cda843f7bc138c74ef574202, c5506b5c58aa4191be250c176296a85f; no voters:
I20260927 14:56:54.762025 2635 raft_consensus.cc:2781] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:56:54.762248 2635 raft_consensus.cc:686] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202 [term 1 LEADER]: Becoming Leader. State: Replica: 44d7da27cda843f7bc138c74ef574202, State: Running, Role: LEADER
I20260927 14:56:54.762621 2635 consensus_queue.cc:237] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:54.765169 2652 tablet_replica.cc:439] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202: TxnStatusTablet state changed. Reason: New leader 44d7da27cda843f7bc138c74ef574202. Latest consensus state: current_term: 1 leader_uuid: "44d7da27cda843f7bc138c74ef574202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } } }
I20260927 14:56:54.765333 2652 tablet_replica.cc:442] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202: This TxnStatusTablet replica's current role is: LEADER
I20260927 14:56:54.765618 2654 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 14:56:54.765820 2654 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 14:56:54.766176 2123 catalog_manager.cc:5495] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d7da27cda843f7bc138c74ef574202 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "44d7da27cda843f7bc138c74ef574202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } health_report { overall_health: UNKNOWN } } }
I20260927 14:56:54.793645 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.789849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42795 (local address 127.0.0.1:32926)
0927 14:56:54.790011 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:54.790027 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:54.790147 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:54.790526 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:54.790529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:54.790538 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 14:56:54.790646 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:54.790653 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.791454 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.791458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:54.792063 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:54.792072 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.792496 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.792500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:54.792587 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.792633 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 14:56:54.793315 (+ 682us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 14:56:54.793331 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 14:56:54.793396 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 14:56:54.793494 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 14:56:54.793645 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.789939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32926 (local address 127.0.98.1:42795)
0927 14:56:54.790093 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:54.790097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:54.790112 (+ 15us) server_negotiation.cc:407] Connection header received
0927 14:56:54.790334 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:54.790337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:54.790399 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:54.790485 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:56:54.790761 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.791356 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.792177 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.792392 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.792480 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.792775 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:56:54.793219 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:56:54.793322 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:54.793408 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 14:56:54.793467 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 14:56:54.799444 2560 raft_consensus.cc:1261] T 7341dacdc8df4c579080d1e6842dd266 P d19f76cccc1c4fe192f0503263707fae [term 1 FOLLOWER]: Refusing update from remote peer 44d7da27cda843f7bc138c74ef574202: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:56:54.799443 2420 raft_consensus.cc:1261] T 7341dacdc8df4c579080d1e6842dd266 P c5506b5c58aa4191be250c176296a85f [term 1 FOLLOWER]: Refusing update from remote peer 44d7da27cda843f7bc138c74ef574202: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:56:54.800133 2652 consensus_queue.cc:1035] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:56:54.800307 2635 consensus_queue.cc:1035] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:56:54.803387 2637 tablet_replica.cc:439] T 7341dacdc8df4c579080d1e6842dd266 P c5506b5c58aa4191be250c176296a85f: TxnStatusTablet state changed. Reason: New leader 44d7da27cda843f7bc138c74ef574202. Latest consensus state: current_term: 1 leader_uuid: "44d7da27cda843f7bc138c74ef574202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } } }
I20260927 14:56:54.803511 2637 tablet_replica.cc:442] T 7341dacdc8df4c579080d1e6842dd266 P c5506b5c58aa4191be250c176296a85f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:56:54.803972 2636 tablet_replica.cc:439] T 7341dacdc8df4c579080d1e6842dd266 P d19f76cccc1c4fe192f0503263707fae: TxnStatusTablet state changed. Reason: New leader 44d7da27cda843f7bc138c74ef574202. Latest consensus state: current_term: 1 leader_uuid: "44d7da27cda843f7bc138c74ef574202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } } }
I20260927 14:56:54.804103 2636 tablet_replica.cc:442] T 7341dacdc8df4c579080d1e6842dd266 P d19f76cccc1c4fe192f0503263707fae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:56:54.806316 2635 tablet_replica.cc:439] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "44d7da27cda843f7bc138c74ef574202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } } }
I20260927 14:56:54.806447 2635 tablet_replica.cc:442] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202: This TxnStatusTablet replica's current role is: LEADER
I20260927 14:56:54.806548 2657 mvcc.cc:205] Tried to move back new op lower bound from 7333974076611751936 to 7333974076471062528. Current Snapshot: MvccSnapshot[applied={T|T < 7333974076611751936}]
I20260927 14:56:54.806324 2652 tablet_replica.cc:439] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "44d7da27cda843f7bc138c74ef574202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } } }
I20260927 14:56:54.807395 2652 tablet_replica.cc:442] T 7341dacdc8df4c579080d1e6842dd266 P 44d7da27cda843f7bc138c74ef574202: This TxnStatusTablet replica's current role is: LEADER
I20260927 14:56:54.807713 2636 tablet_replica.cc:439] T 7341dacdc8df4c579080d1e6842dd266 P d19f76cccc1c4fe192f0503263707fae: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "44d7da27cda843f7bc138c74ef574202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } } }
I20260927 14:56:54.807816 2636 tablet_replica.cc:442] T 7341dacdc8df4c579080d1e6842dd266 P d19f76cccc1c4fe192f0503263707fae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:56:54.807950 2637 tablet_replica.cc:439] T 7341dacdc8df4c579080d1e6842dd266 P c5506b5c58aa4191be250c176296a85f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "44d7da27cda843f7bc138c74ef574202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } } }
I20260927 14:56:54.808058 2637 tablet_replica.cc:442] T 7341dacdc8df4c579080d1e6842dd266 P c5506b5c58aa4191be250c176296a85f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:56:54.809192 2659 mvcc.cc:205] Tried to move back new op lower bound from 7333974076611751936 to 7333974076471062528. Current Snapshot: MvccSnapshot[applied={T|T < 7333974076611751936}]
I20260927 14:56:54.809475 2658 mvcc.cc:205] Tried to move back new op lower bound from 7333974076611751936 to 7333974076471062528. Current Snapshot: MvccSnapshot[applied={T|T < 7333974076611751936}]
I20260927 14:56:54.820116 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.816220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33088 (local address 127.0.98.2:35587)
0927 14:56:54.816342 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:54.816352 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:54.816375 (+ 23us) server_negotiation.cc:407] Connection header received
0927 14:56:54.816569 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:54.816573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:54.816638 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:54.816751 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:56:54.816964 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.817737 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.818550 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.818909 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.819003 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.819217 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:56:54.819773 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:56:54.819884 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:54.819915 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 14:56:54.819973 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 14:56:54.830932 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.827400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42795 (local address 127.0.98.2:60179)
0927 14:56:54.827677 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:54.827691 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:54.827789 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:54.828024 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:54.828026 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:54.828032 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 14:56:54.828135 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:54.828139 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.829086 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.829087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:54.829694 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:54.829701 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.830245 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.830248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:54.830312 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.830353 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 14:56:54.830637 (+ 284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 14:56:54.830640 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 14:56:54.830681 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 14:56:54.830744 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":89,"threads_started":1}
I20260927 14:56:54.830922 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.827520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60179 (local address 127.0.98.1:42795)
0927 14:56:54.827654 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:54.827658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:54.827758 (+ 100us) server_negotiation.cc:407] Connection header received
0927 14:56:54.827875 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:54.827878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:54.827941 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:54.828021 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:56:54.828222 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.828996 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.829797 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.830136 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.830234 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.830487 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:56:54.830568 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:56:54.830626 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:54.830698 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 14:56:54.830750 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 14:56:54.838157 2560 raft_consensus.cc:1261] T df3c7a778ab048188bfd51b51d0537ea P d19f76cccc1c4fe192f0503263707fae [term 1 FOLLOWER]: Refusing update from remote peer c5506b5c58aa4191be250c176296a85f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:56:54.838340 2280 raft_consensus.cc:1261] T df3c7a778ab048188bfd51b51d0537ea P 44d7da27cda843f7bc138c74ef574202 [term 1 FOLLOWER]: Refusing update from remote peer c5506b5c58aa4191be250c176296a85f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:56:54.838665 2639 consensus_queue.cc:1035] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:56:54.838862 2637 consensus_queue.cc:1035] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:56:54.863991 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.859657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38875 (local address 127.0.98.2:35587)
0927 14:56:54.859898 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:54.859903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:54.860400 (+ 497us) server_negotiation.cc:407] Connection header received
0927 14:56:54.860566 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:54.860569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:54.860628 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:54.860702 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:56:54.860978 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.861806 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.862854 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.863144 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.863246 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.863517 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:56:54.863601 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:56:54.863659 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:54.863795 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 14:56:54.863853 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260927 14:56:54.864002 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:54.859583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35587 (local address 127.0.98.1:38875)
0927 14:56:54.859897 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:54.859909 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:54.860435 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:54.860740 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:54.860743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:54.860752 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 14:56:54.860872 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:54.860879 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.861914 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.861917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:54.862711 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:54.862722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:54.863242 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:54.863246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:54.863331 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:54.863388 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 14:56:54.863697 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 14:56:54.863703 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 14:56:54.863765 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 14:56:54.863854 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":120,"threads_started":1}
I20260927 14:56:54.892442 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:56:54.892654 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:56:54.895629 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 132a9d51505746b4a773d526a0d7c474: Sending DeleteTablet for 3 replicas of tablet df3c7a778ab048188bfd51b51d0537ea
I20260927 14:56:54.896677 2527 tablet_service.cc:1514] Processing DeleteTablet for tablet df3c7a778ab048188bfd51b51d0537ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:56:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44202
I20260927 14:56:54.896880 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet df3c7a778ab048188bfd51b51d0537ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:56:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32920
I20260927 14:56:54.897245 2678 tablet_replica.cc:330] T df3c7a778ab048188bfd51b51d0537ea P d19f76cccc1c4fe192f0503263707fae: stopping tablet replica
I20260927 14:56:54.897607 2678 raft_consensus.cc:2229] T df3c7a778ab048188bfd51b51d0537ea P d19f76cccc1c4fe192f0503263707fae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:56:54.897732 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet df3c7a778ab048188bfd51b51d0537ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:56:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33084
I20260927 14:56:54.897850 2678 raft_consensus.cc:2258] T df3c7a778ab048188bfd51b51d0537ea P d19f76cccc1c4fe192f0503263707fae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:56:54.899112 2679 tablet_replica.cc:330] T df3c7a778ab048188bfd51b51d0537ea P 44d7da27cda843f7bc138c74ef574202: stopping tablet replica
I20260927 14:56:54.899101 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 14:56:54.899572 2680 tablet_replica.cc:330] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f: stopping tablet replica
W20260927 14:56:54.899570 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:56:54.899982 2680 raft_consensus.cc:2229] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:56:54.900378 2679 raft_consensus.cc:2229] T df3c7a778ab048188bfd51b51d0537ea P 44d7da27cda843f7bc138c74ef574202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:56:54.900640 2679 raft_consensus.cc:2258] T df3c7a778ab048188bfd51b51d0537ea P 44d7da27cda843f7bc138c74ef574202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:56:54.901070 2680 raft_consensus.cc:2258] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:56:54.904098 2680 ts_tablet_manager.cc:1905] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:56:54.902738 2679 ts_tablet_manager.cc:1905] T df3c7a778ab048188bfd51b51d0537ea P 44d7da27cda843f7bc138c74ef574202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:56:54.905591 2527 tablet_service.cc:1467] Processing CreateTablet for tablet ee128ede493c44eeaf37935e6f834a9c (DEFAULT_TABLE table=test-table [id=ef6b7588743a40bda4c9fa3ca40fe0e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:56:54.905638 2260 tablet_service.cc:1467] Processing CreateTablet for tablet ee128ede493c44eeaf37935e6f834a9c (DEFAULT_TABLE table=test-table [id=ef6b7588743a40bda4c9fa3ca40fe0e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:56:54.905898 2527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee128ede493c44eeaf37935e6f834a9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:54.906059 2400 tablet_service.cc:1467] Processing CreateTablet for tablet ee128ede493c44eeaf37935e6f834a9c (DEFAULT_TABLE table=test-table [id=ef6b7588743a40bda4c9fa3ca40fe0e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:56:54.906482 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee128ede493c44eeaf37935e6f834a9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:54.908342 2631 tablet_bootstrap.cc:492] T ee128ede493c44eeaf37935e6f834a9c P d19f76cccc1c4fe192f0503263707fae: Bootstrap starting.
I20260927 14:56:54.909157 2630 tablet_bootstrap.cc:492] T ee128ede493c44eeaf37935e6f834a9c P c5506b5c58aa4191be250c176296a85f: Bootstrap starting.
I20260927 14:56:54.909310 2631 tablet_bootstrap.cc:654] T ee128ede493c44eeaf37935e6f834a9c P d19f76cccc1c4fe192f0503263707fae: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:54.909832 2678 ts_tablet_manager.cc:1905] T df3c7a778ab048188bfd51b51d0537ea P d19f76cccc1c4fe192f0503263707fae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:56:54.910089 2630 tablet_bootstrap.cc:654] T ee128ede493c44eeaf37935e6f834a9c P c5506b5c58aa4191be250c176296a85f: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:54.913655 2680 ts_tablet_manager.cc:1918] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 14:56:54.913928 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee128ede493c44eeaf37935e6f834a9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:54.914547 2678 ts_tablet_manager.cc:1918] T df3c7a778ab048188bfd51b51d0537ea P d19f76cccc1c4fe192f0503263707fae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 14:56:54.914633 2678 log.cc:1196] T df3c7a778ab048188bfd51b51d0537ea P d19f76cccc1c4fe192f0503263707fae: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-2/wal/wals/df3c7a778ab048188bfd51b51d0537ea
I20260927 14:56:54.914970 2678 ts_tablet_manager.cc:1939] T df3c7a778ab048188bfd51b51d0537ea P d19f76cccc1c4fe192f0503263707fae: Deleting consensus metadata
I20260927 14:56:54.916338 2629 tablet_bootstrap.cc:492] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202: Bootstrap starting.
I20260927 14:56:54.917117 2110 catalog_manager.cc:4841] TS d19f76cccc1c4fe192f0503263707fae (127.0.98.3:43043): tablet df3c7a778ab048188bfd51b51d0537ea (table test-table [id=cc47fe58064c4c1d8a97cddad0d0f1fb]) successfully deleted
I20260927 14:56:54.917297 2629 tablet_bootstrap.cc:654] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:54.917852 2631 tablet_bootstrap.cc:492] T ee128ede493c44eeaf37935e6f834a9c P d19f76cccc1c4fe192f0503263707fae: No bootstrap required, opened a new log
I20260927 14:56:54.917949 2631 ts_tablet_manager.cc:1397] T ee128ede493c44eeaf37935e6f834a9c P d19f76cccc1c4fe192f0503263707fae: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260927 14:56:54.918382 2631 raft_consensus.cc:348] T ee128ede493c44eeaf37935e6f834a9c P d19f76cccc1c4fe192f0503263707fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:54.918504 2631 raft_consensus.cc:374] T ee128ede493c44eeaf37935e6f834a9c P d19f76cccc1c4fe192f0503263707fae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:54.918540 2631 raft_consensus.cc:729] T ee128ede493c44eeaf37935e6f834a9c P d19f76cccc1c4fe192f0503263707fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19f76cccc1c4fe192f0503263707fae, State: Initialized, Role: FOLLOWER
I20260927 14:56:54.918651 2631 consensus_queue.cc:260] T ee128ede493c44eeaf37935e6f834a9c P d19f76cccc1c4fe192f0503263707fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:54.918941 2631 ts_tablet_manager.cc:1428] T ee128ede493c44eeaf37935e6f834a9c P d19f76cccc1c4fe192f0503263707fae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:56:54.920930 2679 ts_tablet_manager.cc:1918] T df3c7a778ab048188bfd51b51d0537ea P 44d7da27cda843f7bc138c74ef574202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 14:56:54.921134 2680 log.cc:1196] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-1/wal/wals/df3c7a778ab048188bfd51b51d0537ea
I20260927 14:56:54.921537 2680 ts_tablet_manager.cc:1939] T df3c7a778ab048188bfd51b51d0537ea P c5506b5c58aa4191be250c176296a85f: Deleting consensus metadata
I20260927 14:56:54.921655 2679 log.cc:1196] T df3c7a778ab048188bfd51b51d0537ea P 44d7da27cda843f7bc138c74ef574202: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-0/wal/wals/df3c7a778ab048188bfd51b51d0537ea
I20260927 14:56:54.922672 2679 ts_tablet_manager.cc:1939] T df3c7a778ab048188bfd51b51d0537ea P 44d7da27cda843f7bc138c74ef574202: Deleting consensus metadata
I20260927 14:56:54.922894 2630 tablet_bootstrap.cc:492] T ee128ede493c44eeaf37935e6f834a9c P c5506b5c58aa4191be250c176296a85f: No bootstrap required, opened a new log
I20260927 14:56:54.923002 2630 ts_tablet_manager.cc:1397] T ee128ede493c44eeaf37935e6f834a9c P c5506b5c58aa4191be250c176296a85f: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20260927 14:56:54.923375 2629 tablet_bootstrap.cc:492] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202: No bootstrap required, opened a new log
I20260927 14:56:54.923475 2630 raft_consensus.cc:348] T ee128ede493c44eeaf37935e6f834a9c P c5506b5c58aa4191be250c176296a85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:54.923552 2629 ts_tablet_manager.cc:1397] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260927 14:56:54.923585 2630 raft_consensus.cc:374] T ee128ede493c44eeaf37935e6f834a9c P c5506b5c58aa4191be250c176296a85f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:54.923614 2630 raft_consensus.cc:729] T ee128ede493c44eeaf37935e6f834a9c P c5506b5c58aa4191be250c176296a85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5506b5c58aa4191be250c176296a85f, State: Initialized, Role: FOLLOWER
I20260927 14:56:54.923744 2630 consensus_queue.cc:260] T ee128ede493c44eeaf37935e6f834a9c P c5506b5c58aa4191be250c176296a85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:54.924003 2630 ts_tablet_manager.cc:1428] T ee128ede493c44eeaf37935e6f834a9c P c5506b5c58aa4191be250c176296a85f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 14:56:54.923945 2629 raft_consensus.cc:348] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:54.924049 2629 raft_consensus.cc:374] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:54.924077 2629 raft_consensus.cc:729] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d7da27cda843f7bc138c74ef574202, State: Initialized, Role: FOLLOWER
I20260927 14:56:54.924185 2629 consensus_queue.cc:260] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:54.924436 2629 ts_tablet_manager.cc:1428] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:56:54.927029 2109 catalog_manager.cc:4841] TS 44d7da27cda843f7bc138c74ef574202 (127.0.98.1:42795): tablet df3c7a778ab048188bfd51b51d0537ea (table test-table [id=cc47fe58064c4c1d8a97cddad0d0f1fb]) successfully deleted
I20260927 14:56:54.928350 2109 catalog_manager.cc:4841] TS c5506b5c58aa4191be250c176296a85f (127.0.98.2:35587): tablet df3c7a778ab048188bfd51b51d0537ea (table test-table [id=cc47fe58064c4c1d8a97cddad0d0f1fb]) successfully deleted
I20260927 14:56:55.015539 2652 raft_consensus.cc:482] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:56:55.015684 2652 raft_consensus.cc:504] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:55.016093 2652 leader_election.cc:288] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5506b5c58aa4191be250c176296a85f (127.0.98.2:35587), d19f76cccc1c4fe192f0503263707fae (127.0.98.3:43043)
I20260927 14:56:55.016528 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee128ede493c44eeaf37935e6f834a9c" candidate_uuid: "44d7da27cda843f7bc138c74ef574202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f76cccc1c4fe192f0503263707fae" is_pre_election: true
I20260927 14:56:55.016536 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee128ede493c44eeaf37935e6f834a9c" candidate_uuid: "44d7da27cda843f7bc138c74ef574202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5506b5c58aa4191be250c176296a85f" is_pre_election: true
I20260927 14:56:55.016678 2560 raft_consensus.cc:2454] T ee128ede493c44eeaf37935e6f834a9c P d19f76cccc1c4fe192f0503263707fae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d7da27cda843f7bc138c74ef574202 in term 0.
I20260927 14:56:55.016696 2420 raft_consensus.cc:2454] T ee128ede493c44eeaf37935e6f834a9c P c5506b5c58aa4191be250c176296a85f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d7da27cda843f7bc138c74ef574202 in term 0.
I20260927 14:56:55.016986 2214 leader_election.cc:302] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d7da27cda843f7bc138c74ef574202, d19f76cccc1c4fe192f0503263707fae; no voters:
I20260927 14:56:55.017203 2652 raft_consensus.cc:2781] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:56:55.017272 2652 raft_consensus.cc:482] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:56:55.017324 2652 raft_consensus.cc:3037] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:55.018359 2652 raft_consensus.cc:504] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:55.018698 2652 leader_election.cc:288] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202 [CANDIDATE]: Term 1 election: Requested vote from peers c5506b5c58aa4191be250c176296a85f (127.0.98.2:35587), d19f76cccc1c4fe192f0503263707fae (127.0.98.3:43043)
I20260927 14:56:55.019058 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee128ede493c44eeaf37935e6f834a9c" candidate_uuid: "44d7da27cda843f7bc138c74ef574202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5506b5c58aa4191be250c176296a85f"
I20260927 14:56:55.019095 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee128ede493c44eeaf37935e6f834a9c" candidate_uuid: "44d7da27cda843f7bc138c74ef574202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f76cccc1c4fe192f0503263707fae"
I20260927 14:56:55.019196 2420 raft_consensus.cc:3037] T ee128ede493c44eeaf37935e6f834a9c P c5506b5c58aa4191be250c176296a85f [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:55.019196 2560 raft_consensus.cc:3037] T ee128ede493c44eeaf37935e6f834a9c P d19f76cccc1c4fe192f0503263707fae [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:55.020251 2420 raft_consensus.cc:2454] T ee128ede493c44eeaf37935e6f834a9c P c5506b5c58aa4191be250c176296a85f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d7da27cda843f7bc138c74ef574202 in term 1.
I20260927 14:56:55.020249 2560 raft_consensus.cc:2454] T ee128ede493c44eeaf37935e6f834a9c P d19f76cccc1c4fe192f0503263707fae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d7da27cda843f7bc138c74ef574202 in term 1.
I20260927 14:56:55.020594 2213 leader_election.cc:302] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d7da27cda843f7bc138c74ef574202, c5506b5c58aa4191be250c176296a85f; no voters:
I20260927 14:56:55.020826 2652 raft_consensus.cc:2781] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:56:55.020924 2652 raft_consensus.cc:686] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202 [term 1 LEADER]: Becoming Leader. State: Replica: 44d7da27cda843f7bc138c74ef574202, State: Running, Role: LEADER
I20260927 14:56:55.021060 2652 consensus_queue.cc:237] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:55.022737 2122 catalog_manager.cc:5495] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d7da27cda843f7bc138c74ef574202 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "44d7da27cda843f7bc138c74ef574202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } health_report { overall_health: UNKNOWN } } }
I20260927 14:56:55.052198 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:55.049109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32938 (local address 127.0.98.1:42795)
0927 14:56:55.049226 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:55.049229 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:55.049243 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:56:55.049364 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:55.049366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:55.049410 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:55.049467 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:56:55.049691 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:55.050183 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:55.051334 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:55.051536 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:55.051606 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:55.051841 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:56:55.051908 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:56:55.051952 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:55.052036 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 14:56:55.052081 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:56:55.061249 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:55.058391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58889 (local address 127.0.98.1:42795)
0927 14:56:55.058482 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:55.058486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:55.058532 (+ 46us) server_negotiation.cc:407] Connection header received
0927 14:56:55.058666 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:55.058669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:55.058710 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:55.058767 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:56:55.059003 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:55.059503 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:55.060318 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:55.060532 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:55.060612 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:55.060831 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:56:55.060902 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:56:55.060976 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:55.061045 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 14:56:55.061091 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 14:56:55.061249 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:55.058321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42795 (local address 127.0.98.1:58889)
0927 14:56:55.058471 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:55.058486 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:56:55.058580 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:56:55.058826 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:56:55.058828 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:56:55.058835 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 14:56:55.058918 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:55.058924 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:55.059602 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:55.059605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:55.060216 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:56:55.060225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:56:55.060628 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:56:55.060631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:56:55.060706 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:55.060733 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 14:56:55.060977 (+ 244us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 14:56:55.060982 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 14:56:55.061039 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 14:56:55.061131 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":3}
I20260927 14:56:55.067160 2560 raft_consensus.cc:1261] T ee128ede493c44eeaf37935e6f834a9c P d19f76cccc1c4fe192f0503263707fae [term 1 FOLLOWER]: Refusing update from remote peer 44d7da27cda843f7bc138c74ef574202: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:56:55.067307 2420 raft_consensus.cc:1261] T ee128ede493c44eeaf37935e6f834a9c P c5506b5c58aa4191be250c176296a85f [term 1 FOLLOWER]: Refusing update from remote peer 44d7da27cda843f7bc138c74ef574202: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:56:55.067597 2663 consensus_queue.cc:1035] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:56:55.067764 2635 consensus_queue.cc:1035] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 14:56:55.081455 2657 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 14:56:56.635944 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:56:56.636109 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:56:56.638471 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 132a9d51505746b4a773d526a0d7c474: Sending DeleteTablet for 3 replicas of tablet ee128ede493c44eeaf37935e6f834a9c
I20260927 14:56:56.639154 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet ee128ede493c44eeaf37935e6f834a9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:56:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32920
I20260927 14:56:56.639142 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet ee128ede493c44eeaf37935e6f834a9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:56:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33084
I20260927 14:56:56.639196 2527 tablet_service.cc:1514] Processing DeleteTablet for tablet ee128ede493c44eeaf37935e6f834a9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:56:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44202
I20260927 14:56:56.639771 2710 tablet_replica.cc:330] T ee128ede493c44eeaf37935e6f834a9c P d19f76cccc1c4fe192f0503263707fae: stopping tablet replica
I20260927 14:56:56.639954 2710 raft_consensus.cc:2229] T ee128ede493c44eeaf37935e6f834a9c P d19f76cccc1c4fe192f0503263707fae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:56:56.640151 2710 raft_consensus.cc:2258] T ee128ede493c44eeaf37935e6f834a9c P d19f76cccc1c4fe192f0503263707fae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:56:56.640357 2709 tablet_replica.cc:330] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202: stopping tablet replica
I20260927 14:56:56.640486 2709 raft_consensus.cc:2229] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202 [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:56:56.640813 2709 raft_consensus.cc:2258] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:56:56.641091 2710 ts_tablet_manager.cc:1905] T ee128ede493c44eeaf37935e6f834a9c P d19f76cccc1c4fe192f0503263707fae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:56:56.641458 2709 ts_tablet_manager.cc:1905] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:56:56.641646 2708 tablet_replica.cc:330] T ee128ede493c44eeaf37935e6f834a9c P c5506b5c58aa4191be250c176296a85f: stopping tablet replica
I20260927 14:56:56.641786 2708 raft_consensus.cc:2229] T ee128ede493c44eeaf37935e6f834a9c P c5506b5c58aa4191be250c176296a85f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:56:56.641914 2708 raft_consensus.cc:2258] T ee128ede493c44eeaf37935e6f834a9c P c5506b5c58aa4191be250c176296a85f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:56:56.642470 2708 ts_tablet_manager.cc:1905] T ee128ede493c44eeaf37935e6f834a9c P c5506b5c58aa4191be250c176296a85f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:56:56.644034 2709 ts_tablet_manager.cc:1918] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 14:56:56.644034 2710 ts_tablet_manager.cc:1918] T ee128ede493c44eeaf37935e6f834a9c P d19f76cccc1c4fe192f0503263707fae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 14:56:56.644114 2710 log.cc:1196] T ee128ede493c44eeaf37935e6f834a9c P d19f76cccc1c4fe192f0503263707fae: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-2/wal/wals/ee128ede493c44eeaf37935e6f834a9c
I20260927 14:56:56.644114 2709 log.cc:1196] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-0/wal/wals/ee128ede493c44eeaf37935e6f834a9c
I20260927 14:56:56.644383 2709 ts_tablet_manager.cc:1939] T ee128ede493c44eeaf37935e6f834a9c P 44d7da27cda843f7bc138c74ef574202: Deleting consensus metadata
I20260927 14:56:56.644394 2710 ts_tablet_manager.cc:1939] T ee128ede493c44eeaf37935e6f834a9c P d19f76cccc1c4fe192f0503263707fae: Deleting consensus metadata
I20260927 14:56:56.644886 2708 ts_tablet_manager.cc:1918] T ee128ede493c44eeaf37935e6f834a9c P c5506b5c58aa4191be250c176296a85f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 14:56:56.644953 2708 log.cc:1196] T ee128ede493c44eeaf37935e6f834a9c P c5506b5c58aa4191be250c176296a85f: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-1/wal/wals/ee128ede493c44eeaf37935e6f834a9c
I20260927 14:56:56.645129 2110 catalog_manager.cc:4841] TS d19f76cccc1c4fe192f0503263707fae (127.0.98.3:43043): tablet ee128ede493c44eeaf37935e6f834a9c (table test-table [id=ef6b7588743a40bda4c9fa3ca40fe0e1]) successfully deleted
I20260927 14:56:56.645175 2708 ts_tablet_manager.cc:1939] T ee128ede493c44eeaf37935e6f834a9c P c5506b5c58aa4191be250c176296a85f: Deleting consensus metadata
I20260927 14:56:56.645238 2109 catalog_manager.cc:4841] TS 44d7da27cda843f7bc138c74ef574202 (127.0.98.1:42795): tablet ee128ede493c44eeaf37935e6f834a9c (table test-table [id=ef6b7588743a40bda4c9fa3ca40fe0e1]) successfully deleted
I20260927 14:56:56.645783 2109 catalog_manager.cc:4841] TS c5506b5c58aa4191be250c176296a85f (127.0.98.2:35587): tablet ee128ede493c44eeaf37935e6f834a9c (table test-table [id=ef6b7588743a40bda4c9fa3ca40fe0e1]) successfully deleted
Sep 27 14:56:56 dist-test-slave-pz0h krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521016, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 14:56:56 dist-test-slave-pz0h krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521016, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:56:56.674940 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:56.659764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50388 (local address 127.0.98.62:40439)
0927 14:56:56.660012 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:56.660018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:56.660068 (+ 50us) server_negotiation.cc:407] Connection header received
0927 14:56:56.660196 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:56.660198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:56.660242 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:56.660344 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:56.661024 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:56.661599 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:56.662683 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:56.662952 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:56.663031 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:56.672032 (+ 9001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:56.672036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:56.672039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:56.672081 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:56.673471 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:56.673867 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:56.673869 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:56.673870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:56.673945 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:56.674134 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:56.674136 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:56.674137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:56.674321 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:56.674430 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:56.674653 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 14:56:56.674757 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":89,"threads_started":1}
I20260927 14:56:56.677346 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:56:56.677734 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:56:56.682013 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 37c0ed28630140579309b338b4147019 (DEFAULT_TABLE table=test-table [id=0d7f984dd8404262910ae761b2d99dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:56:56.681983 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 37c0ed28630140579309b338b4147019 (DEFAULT_TABLE table=test-table [id=0d7f984dd8404262910ae761b2d99dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:56:56.682025 2527 tablet_service.cc:1467] Processing CreateTablet for tablet 37c0ed28630140579309b338b4147019 (DEFAULT_TABLE table=test-table [id=0d7f984dd8404262910ae761b2d99dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:56:56.682314 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c0ed28630140579309b338b4147019. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:56.682314 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c0ed28630140579309b338b4147019. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:56.682350 2527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c0ed28630140579309b338b4147019. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:56.684494 2719 tablet_bootstrap.cc:492] T 37c0ed28630140579309b338b4147019 P d19f76cccc1c4fe192f0503263707fae: Bootstrap starting.
I20260927 14:56:56.684877 2721 tablet_bootstrap.cc:492] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202: Bootstrap starting.
I20260927 14:56:56.685328 2720 tablet_bootstrap.cc:492] T 37c0ed28630140579309b338b4147019 P c5506b5c58aa4191be250c176296a85f: Bootstrap starting.
I20260927 14:56:56.685547 2719 tablet_bootstrap.cc:654] T 37c0ed28630140579309b338b4147019 P d19f76cccc1c4fe192f0503263707fae: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:56.685746 2721 tablet_bootstrap.cc:654] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:56.686241 2720 tablet_bootstrap.cc:654] T 37c0ed28630140579309b338b4147019 P c5506b5c58aa4191be250c176296a85f: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:56.686771 2719 tablet_bootstrap.cc:492] T 37c0ed28630140579309b338b4147019 P d19f76cccc1c4fe192f0503263707fae: No bootstrap required, opened a new log
I20260927 14:56:56.686889 2719 ts_tablet_manager.cc:1397] T 37c0ed28630140579309b338b4147019 P d19f76cccc1c4fe192f0503263707fae: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 14:56:56.686920 2721 tablet_bootstrap.cc:492] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202: No bootstrap required, opened a new log
I20260927 14:56:56.687016 2721 ts_tablet_manager.cc:1397] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 14:56:56.687371 2720 tablet_bootstrap.cc:492] T 37c0ed28630140579309b338b4147019 P c5506b5c58aa4191be250c176296a85f: No bootstrap required, opened a new log
I20260927 14:56:56.687362 2719 raft_consensus.cc:348] T 37c0ed28630140579309b338b4147019 P d19f76cccc1c4fe192f0503263707fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:56.687460 2720 ts_tablet_manager.cc:1397] T 37c0ed28630140579309b338b4147019 P c5506b5c58aa4191be250c176296a85f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 14:56:56.687490 2719 raft_consensus.cc:374] T 37c0ed28630140579309b338b4147019 P d19f76cccc1c4fe192f0503263707fae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:56.687528 2719 raft_consensus.cc:729] T 37c0ed28630140579309b338b4147019 P d19f76cccc1c4fe192f0503263707fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19f76cccc1c4fe192f0503263707fae, State: Initialized, Role: FOLLOWER
I20260927 14:56:56.687657 2719 consensus_queue.cc:260] T 37c0ed28630140579309b338b4147019 P d19f76cccc1c4fe192f0503263707fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:56.687824 2720 raft_consensus.cc:348] T 37c0ed28630140579309b338b4147019 P c5506b5c58aa4191be250c176296a85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:56.687925 2720 raft_consensus.cc:374] T 37c0ed28630140579309b338b4147019 P c5506b5c58aa4191be250c176296a85f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:56.687958 2720 raft_consensus.cc:729] T 37c0ed28630140579309b338b4147019 P c5506b5c58aa4191be250c176296a85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5506b5c58aa4191be250c176296a85f, State: Initialized, Role: FOLLOWER
I20260927 14:56:56.688045 2719 ts_tablet_manager.cc:1428] T 37c0ed28630140579309b338b4147019 P d19f76cccc1c4fe192f0503263707fae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:56:56.688051 2721 raft_consensus.cc:348] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:56.688094 2720 consensus_queue.cc:260] T 37c0ed28630140579309b338b4147019 P c5506b5c58aa4191be250c176296a85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:56.688165 2721 raft_consensus.cc:374] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:56.688196 2721 raft_consensus.cc:729] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d7da27cda843f7bc138c74ef574202, State: Initialized, Role: FOLLOWER
I20260927 14:56:56.688310 2721 consensus_queue.cc:260] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:56.688423 2720 ts_tablet_manager.cc:1428] T 37c0ed28630140579309b338b4147019 P c5506b5c58aa4191be250c176296a85f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:56:56.688555 2721 ts_tablet_manager.cc:1428] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 14:56:56.774302 2663 raft_consensus.cc:482] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:56:56.774489 2663 raft_consensus.cc:504] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:56.775054 2663 leader_election.cc:288] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5506b5c58aa4191be250c176296a85f (127.0.98.2:35587), d19f76cccc1c4fe192f0503263707fae (127.0.98.3:43043)
I20260927 14:56:56.775518 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c0ed28630140579309b338b4147019" candidate_uuid: "44d7da27cda843f7bc138c74ef574202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5506b5c58aa4191be250c176296a85f" is_pre_election: true
I20260927 14:56:56.775539 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c0ed28630140579309b338b4147019" candidate_uuid: "44d7da27cda843f7bc138c74ef574202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f76cccc1c4fe192f0503263707fae" is_pre_election: true
I20260927 14:56:56.775664 2559 raft_consensus.cc:2454] T 37c0ed28630140579309b338b4147019 P d19f76cccc1c4fe192f0503263707fae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d7da27cda843f7bc138c74ef574202 in term 0.
I20260927 14:56:56.775667 2419 raft_consensus.cc:2454] T 37c0ed28630140579309b338b4147019 P c5506b5c58aa4191be250c176296a85f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d7da27cda843f7bc138c74ef574202 in term 0.
I20260927 14:56:56.775997 2214 leader_election.cc:302] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d7da27cda843f7bc138c74ef574202, d19f76cccc1c4fe192f0503263707fae; no voters:
I20260927 14:56:56.776173 2663 raft_consensus.cc:2781] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:56:56.776234 2663 raft_consensus.cc:482] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:56:56.776279 2663 raft_consensus.cc:3037] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:56.777268 2663 raft_consensus.cc:504] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:56.778155 2663 leader_election.cc:288] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202 [CANDIDATE]: Term 1 election: Requested vote from peers c5506b5c58aa4191be250c176296a85f (127.0.98.2:35587), d19f76cccc1c4fe192f0503263707fae (127.0.98.3:43043)
I20260927 14:56:56.778582 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c0ed28630140579309b338b4147019" candidate_uuid: "44d7da27cda843f7bc138c74ef574202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5506b5c58aa4191be250c176296a85f"
I20260927 14:56:56.778731 2419 raft_consensus.cc:3037] T 37c0ed28630140579309b338b4147019 P c5506b5c58aa4191be250c176296a85f [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:56.778889 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c0ed28630140579309b338b4147019" candidate_uuid: "44d7da27cda843f7bc138c74ef574202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f76cccc1c4fe192f0503263707fae"
I20260927 14:56:56.779239 2559 raft_consensus.cc:3037] T 37c0ed28630140579309b338b4147019 P d19f76cccc1c4fe192f0503263707fae [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:56.779843 2419 raft_consensus.cc:2454] T 37c0ed28630140579309b338b4147019 P c5506b5c58aa4191be250c176296a85f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d7da27cda843f7bc138c74ef574202 in term 1.
I20260927 14:56:56.780187 2213 leader_election.cc:302] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d7da27cda843f7bc138c74ef574202, c5506b5c58aa4191be250c176296a85f; no voters:
I20260927 14:56:56.780345 2559 raft_consensus.cc:2454] T 37c0ed28630140579309b338b4147019 P d19f76cccc1c4fe192f0503263707fae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d7da27cda843f7bc138c74ef574202 in term 1.
I20260927 14:56:56.780382 2663 raft_consensus.cc:2781] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:56:56.780470 2663 raft_consensus.cc:686] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202 [term 1 LEADER]: Becoming Leader. State: Replica: 44d7da27cda843f7bc138c74ef574202, State: Running, Role: LEADER
I20260927 14:56:56.780617 2663 consensus_queue.cc:237] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:56.782472 2118 catalog_manager.cc:5495] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d7da27cda843f7bc138c74ef574202 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "44d7da27cda843f7bc138c74ef574202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } health_report { overall_health: UNKNOWN } } }
I20260927 14:56:56.819746 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:56.816488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32948 (local address 127.0.98.1:42795)
0927 14:56:56.816735 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:56.816742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:56.816754 (+ 12us) server_negotiation.cc:407] Connection header received
0927 14:56:56.816803 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:56.816805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:56.816876 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:56.816952 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:56:56.817236 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:56.817755 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:56.818678 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:56.819013 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:56.819111 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:56.819308 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:56:56.819403 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:56:56.819464 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:56.819565 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 14:56:56.819617 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":99,"threads_started":1}
I20260927 14:56:56.826020 2419 raft_consensus.cc:1261] T 37c0ed28630140579309b338b4147019 P c5506b5c58aa4191be250c176296a85f [term 1 FOLLOWER]: Refusing update from remote peer 44d7da27cda843f7bc138c74ef574202: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:56:56.826063 2559 raft_consensus.cc:1261] T 37c0ed28630140579309b338b4147019 P d19f76cccc1c4fe192f0503263707fae [term 1 FOLLOWER]: Refusing update from remote peer 44d7da27cda843f7bc138c74ef574202: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:56:56.826356 2691 consensus_queue.cc:1035] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:56:56.826509 2663 consensus_queue.cc:1035] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:56:56.860917 2123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:56:56.861263 2123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:56:56.863659 2123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 132a9d51505746b4a773d526a0d7c474: Sending DeleteTablet for 3 replicas of tablet 37c0ed28630140579309b338b4147019
I20260927 14:56:56.864295 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 37c0ed28630140579309b338b4147019 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:56:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32920
I20260927 14:56:56.864331 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet 37c0ed28630140579309b338b4147019 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:56:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33084
I20260927 14:56:56.864348 2527 tablet_service.cc:1514] Processing DeleteTablet for tablet 37c0ed28630140579309b338b4147019 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:56:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44202
I20260927 14:56:56.864500 2709 tablet_replica.cc:330] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202: stopping tablet replica
I20260927 14:56:56.864528 2710 tablet_replica.cc:330] T 37c0ed28630140579309b338b4147019 P d19f76cccc1c4fe192f0503263707fae: stopping tablet replica
I20260927 14:56:56.864523 2708 tablet_replica.cc:330] T 37c0ed28630140579309b338b4147019 P c5506b5c58aa4191be250c176296a85f: stopping tablet replica
I20260927 14:56:56.864614 2709 raft_consensus.cc:2229] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202 [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:56:56.864629 2708 raft_consensus.cc:2229] T 37c0ed28630140579309b338b4147019 P c5506b5c58aa4191be250c176296a85f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:56:56.864629 2710 raft_consensus.cc:2229] T 37c0ed28630140579309b338b4147019 P d19f76cccc1c4fe192f0503263707fae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:56:56.864737 2708 raft_consensus.cc:2258] T 37c0ed28630140579309b338b4147019 P c5506b5c58aa4191be250c176296a85f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:56:56.864749 2710 raft_consensus.cc:2258] T 37c0ed28630140579309b338b4147019 P d19f76cccc1c4fe192f0503263707fae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:56:56.864806 2709 raft_consensus.cc:2258] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:56:56.865257 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 14:56:56.865542 2710 ts_tablet_manager.cc:1905] T 37c0ed28630140579309b338b4147019 P d19f76cccc1c4fe192f0503263707fae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:56:56.865541 2708 ts_tablet_manager.cc:1905] T 37c0ed28630140579309b338b4147019 P c5506b5c58aa4191be250c176296a85f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:56:56.865609 2709 ts_tablet_manager.cc:1905] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 14:56:56.865697 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:56:56.867923 2709 ts_tablet_manager.cc:1918] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 14:56:56.867978 2709 log.cc:1196] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-0/wal/wals/37c0ed28630140579309b338b4147019
I20260927 14:56:56.868093 2708 ts_tablet_manager.cc:1918] T 37c0ed28630140579309b338b4147019 P c5506b5c58aa4191be250c176296a85f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 14:56:56.868093 2710 ts_tablet_manager.cc:1918] T 37c0ed28630140579309b338b4147019 P d19f76cccc1c4fe192f0503263707fae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 14:56:56.868140 2709 ts_tablet_manager.cc:1939] T 37c0ed28630140579309b338b4147019 P 44d7da27cda843f7bc138c74ef574202: Deleting consensus metadata
I20260927 14:56:56.868161 2710 log.cc:1196] T 37c0ed28630140579309b338b4147019 P d19f76cccc1c4fe192f0503263707fae: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-2/wal/wals/37c0ed28630140579309b338b4147019
I20260927 14:56:56.868161 2708 log.cc:1196] T 37c0ed28630140579309b338b4147019 P c5506b5c58aa4191be250c176296a85f: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-1/wal/wals/37c0ed28630140579309b338b4147019
I20260927 14:56:56.868391 2708 ts_tablet_manager.cc:1939] T 37c0ed28630140579309b338b4147019 P c5506b5c58aa4191be250c176296a85f: Deleting consensus metadata
I20260927 14:56:56.868391 2710 ts_tablet_manager.cc:1939] T 37c0ed28630140579309b338b4147019 P d19f76cccc1c4fe192f0503263707fae: Deleting consensus metadata
I20260927 14:56:56.869238 2110 catalog_manager.cc:4841] TS d19f76cccc1c4fe192f0503263707fae (127.0.98.3:43043): tablet 37c0ed28630140579309b338b4147019 (table test-table [id=0d7f984dd8404262910ae761b2d99dba]) successfully deleted
I20260927 14:56:56.869240 2109 catalog_manager.cc:4841] TS c5506b5c58aa4191be250c176296a85f (127.0.98.2:35587): tablet 37c0ed28630140579309b338b4147019 (table test-table [id=0d7f984dd8404262910ae761b2d99dba]) successfully deleted
I20260927 14:56:56.869686 2109 catalog_manager.cc:4841] TS 44d7da27cda843f7bc138c74ef574202 (127.0.98.1:42795): tablet 37c0ed28630140579309b338b4147019 (table test-table [id=0d7f984dd8404262910ae761b2d99dba]) successfully deleted
I20260927 14:56:56.872792 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 970a01413d4e4d26846dec7cf500062e (DEFAULT_TABLE table=test-table [id=a09761737933415fadfe63776853fbe0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:56:56.872859 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 970a01413d4e4d26846dec7cf500062e (DEFAULT_TABLE table=test-table [id=a09761737933415fadfe63776853fbe0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:56:56.873014 2527 tablet_service.cc:1467] Processing CreateTablet for tablet 970a01413d4e4d26846dec7cf500062e (DEFAULT_TABLE table=test-table [id=a09761737933415fadfe63776853fbe0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:56:56.873080 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 970a01413d4e4d26846dec7cf500062e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:56.873081 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 970a01413d4e4d26846dec7cf500062e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:56.873263 2527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 970a01413d4e4d26846dec7cf500062e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:56.874994 2719 tablet_bootstrap.cc:492] T 970a01413d4e4d26846dec7cf500062e P d19f76cccc1c4fe192f0503263707fae: Bootstrap starting.
I20260927 14:56:56.875319 2721 tablet_bootstrap.cc:492] T 970a01413d4e4d26846dec7cf500062e P 44d7da27cda843f7bc138c74ef574202: Bootstrap starting.
I20260927 14:56:56.875334 2720 tablet_bootstrap.cc:492] T 970a01413d4e4d26846dec7cf500062e P c5506b5c58aa4191be250c176296a85f: Bootstrap starting.
I20260927 14:56:56.875927 2719 tablet_bootstrap.cc:654] T 970a01413d4e4d26846dec7cf500062e P d19f76cccc1c4fe192f0503263707fae: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:56.875979 2721 tablet_bootstrap.cc:654] T 970a01413d4e4d26846dec7cf500062e P 44d7da27cda843f7bc138c74ef574202: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:56.876053 2720 tablet_bootstrap.cc:654] T 970a01413d4e4d26846dec7cf500062e P c5506b5c58aa4191be250c176296a85f: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:56.877012 2719 tablet_bootstrap.cc:492] T 970a01413d4e4d26846dec7cf500062e P d19f76cccc1c4fe192f0503263707fae: No bootstrap required, opened a new log
I20260927 14:56:56.877012 2721 tablet_bootstrap.cc:492] T 970a01413d4e4d26846dec7cf500062e P 44d7da27cda843f7bc138c74ef574202: No bootstrap required, opened a new log
I20260927 14:56:56.877103 2719 ts_tablet_manager.cc:1397] T 970a01413d4e4d26846dec7cf500062e P d19f76cccc1c4fe192f0503263707fae: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 14:56:56.877103 2721 ts_tablet_manager.cc:1397] T 970a01413d4e4d26846dec7cf500062e P 44d7da27cda843f7bc138c74ef574202: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 14:56:56.877278 2720 tablet_bootstrap.cc:492] T 970a01413d4e4d26846dec7cf500062e P c5506b5c58aa4191be250c176296a85f: No bootstrap required, opened a new log
I20260927 14:56:56.877372 2720 ts_tablet_manager.cc:1397] T 970a01413d4e4d26846dec7cf500062e P c5506b5c58aa4191be250c176296a85f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 14:56:56.877537 2721 raft_consensus.cc:348] T 970a01413d4e4d26846dec7cf500062e P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:56.877537 2719 raft_consensus.cc:348] T 970a01413d4e4d26846dec7cf500062e P d19f76cccc1c4fe192f0503263707fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:56.877657 2719 raft_consensus.cc:374] T 970a01413d4e4d26846dec7cf500062e P d19f76cccc1c4fe192f0503263707fae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:56.877657 2721 raft_consensus.cc:374] T 970a01413d4e4d26846dec7cf500062e P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:56.877699 2721 raft_consensus.cc:729] T 970a01413d4e4d26846dec7cf500062e P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d7da27cda843f7bc138c74ef574202, State: Initialized, Role: FOLLOWER
I20260927 14:56:56.877699 2719 raft_consensus.cc:729] T 970a01413d4e4d26846dec7cf500062e P d19f76cccc1c4fe192f0503263707fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19f76cccc1c4fe192f0503263707fae, State: Initialized, Role: FOLLOWER
I20260927 14:56:56.877729 2720 raft_consensus.cc:348] T 970a01413d4e4d26846dec7cf500062e P c5506b5c58aa4191be250c176296a85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:56.877842 2720 raft_consensus.cc:374] T 970a01413d4e4d26846dec7cf500062e P c5506b5c58aa4191be250c176296a85f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:56.877820 2721 consensus_queue.cc:260] T 970a01413d4e4d26846dec7cf500062e P 44d7da27cda843f7bc138c74ef574202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:56.877820 2719 consensus_queue.cc:260] T 970a01413d4e4d26846dec7cf500062e P d19f76cccc1c4fe192f0503263707fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:56.877879 2720 raft_consensus.cc:729] T 970a01413d4e4d26846dec7cf500062e P c5506b5c58aa4191be250c176296a85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5506b5c58aa4191be250c176296a85f, State: Initialized, Role: FOLLOWER
I20260927 14:56:56.877995 2720 consensus_queue.cc:260] T 970a01413d4e4d26846dec7cf500062e P c5506b5c58aa4191be250c176296a85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:56.878070 2719 ts_tablet_manager.cc:1428] T 970a01413d4e4d26846dec7cf500062e P d19f76cccc1c4fe192f0503263707fae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:56:56.878070 2721 ts_tablet_manager.cc:1428] T 970a01413d4e4d26846dec7cf500062e P 44d7da27cda843f7bc138c74ef574202: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:56:56.878226 2720 ts_tablet_manager.cc:1428] T 970a01413d4e4d26846dec7cf500062e P c5506b5c58aa4191be250c176296a85f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:56:57.043062 2663 raft_consensus.cc:482] T 970a01413d4e4d26846dec7cf500062e P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:56:57.043219 2663 raft_consensus.cc:504] T 970a01413d4e4d26846dec7cf500062e P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:57.043648 2663 leader_election.cc:288] T 970a01413d4e4d26846dec7cf500062e P 44d7da27cda843f7bc138c74ef574202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5506b5c58aa4191be250c176296a85f (127.0.98.2:35587), d19f76cccc1c4fe192f0503263707fae (127.0.98.3:43043)
I20260927 14:56:57.044201 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970a01413d4e4d26846dec7cf500062e" candidate_uuid: "44d7da27cda843f7bc138c74ef574202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5506b5c58aa4191be250c176296a85f" is_pre_election: true
I20260927 14:56:57.044231 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970a01413d4e4d26846dec7cf500062e" candidate_uuid: "44d7da27cda843f7bc138c74ef574202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f76cccc1c4fe192f0503263707fae" is_pre_election: true
I20260927 14:56:57.044368 2559 raft_consensus.cc:2454] T 970a01413d4e4d26846dec7cf500062e P d19f76cccc1c4fe192f0503263707fae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d7da27cda843f7bc138c74ef574202 in term 0.
I20260927 14:56:57.044375 2420 raft_consensus.cc:2454] T 970a01413d4e4d26846dec7cf500062e P c5506b5c58aa4191be250c176296a85f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d7da27cda843f7bc138c74ef574202 in term 0.
I20260927 14:56:57.044699 2214 leader_election.cc:302] T 970a01413d4e4d26846dec7cf500062e P 44d7da27cda843f7bc138c74ef574202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d7da27cda843f7bc138c74ef574202, d19f76cccc1c4fe192f0503263707fae; no voters:
I20260927 14:56:57.044895 2663 raft_consensus.cc:2781] T 970a01413d4e4d26846dec7cf500062e P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:56:57.044952 2663 raft_consensus.cc:482] T 970a01413d4e4d26846dec7cf500062e P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:56:57.044977 2663 raft_consensus.cc:3037] T 970a01413d4e4d26846dec7cf500062e P 44d7da27cda843f7bc138c74ef574202 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:57.045962 2663 raft_consensus.cc:504] T 970a01413d4e4d26846dec7cf500062e P 44d7da27cda843f7bc138c74ef574202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:57.046314 2663 leader_election.cc:288] T 970a01413d4e4d26846dec7cf500062e P 44d7da27cda843f7bc138c74ef574202 [CANDIDATE]: Term 1 election: Requested vote from peers c5506b5c58aa4191be250c176296a85f (127.0.98.2:35587), d19f76cccc1c4fe192f0503263707fae (127.0.98.3:43043)
I20260927 14:56:57.046694 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970a01413d4e4d26846dec7cf500062e" candidate_uuid: "44d7da27cda843f7bc138c74ef574202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5506b5c58aa4191be250c176296a85f"
I20260927 14:56:57.046701 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970a01413d4e4d26846dec7cf500062e" candidate_uuid: "44d7da27cda843f7bc138c74ef574202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f76cccc1c4fe192f0503263707fae"
I20260927 14:56:57.046836 2420 raft_consensus.cc:3037] T 970a01413d4e4d26846dec7cf500062e P c5506b5c58aa4191be250c176296a85f [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:57.046861 2559 raft_consensus.cc:3037] T 970a01413d4e4d26846dec7cf500062e P d19f76cccc1c4fe192f0503263707fae [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:57.047734 2420 raft_consensus.cc:2454] T 970a01413d4e4d26846dec7cf500062e P c5506b5c58aa4191be250c176296a85f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d7da27cda843f7bc138c74ef574202 in term 1.
I20260927 14:56:57.047765 2559 raft_consensus.cc:2454] T 970a01413d4e4d26846dec7cf500062e P d19f76cccc1c4fe192f0503263707fae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d7da27cda843f7bc138c74ef574202 in term 1.
I20260927 14:56:57.048029 2213 leader_election.cc:302] T 970a01413d4e4d26846dec7cf500062e P 44d7da27cda843f7bc138c74ef574202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d7da27cda843f7bc138c74ef574202, c5506b5c58aa4191be250c176296a85f; no voters:
I20260927 14:56:57.048230 2663 raft_consensus.cc:2781] T 970a01413d4e4d26846dec7cf500062e P 44d7da27cda843f7bc138c74ef574202 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:56:57.048318 2663 raft_consensus.cc:686] T 970a01413d4e4d26846dec7cf500062e P 44d7da27cda843f7bc138c74ef574202 [term 1 LEADER]: Becoming Leader. State: Replica: 44d7da27cda843f7bc138c74ef574202, State: Running, Role: LEADER
I20260927 14:56:57.048450 2663 consensus_queue.cc:237] T 970a01413d4e4d26846dec7cf500062e P 44d7da27cda843f7bc138c74ef574202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } }
I20260927 14:56:57.050014 2123 catalog_manager.cc:5495] T 970a01413d4e4d26846dec7cf500062e P 44d7da27cda843f7bc138c74ef574202 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d7da27cda843f7bc138c74ef574202 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "44d7da27cda843f7bc138c74ef574202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d7da27cda843f7bc138c74ef574202" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 } health_report { overall_health: UNKNOWN } } }
I20260927 14:56:57.080673 2420 raft_consensus.cc:1261] T 970a01413d4e4d26846dec7cf500062e P c5506b5c58aa4191be250c176296a85f [term 1 FOLLOWER]: Refusing update from remote peer 44d7da27cda843f7bc138c74ef574202: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:56:57.080761 2559 raft_consensus.cc:1261] T 970a01413d4e4d26846dec7cf500062e P d19f76cccc1c4fe192f0503263707fae [term 1 FOLLOWER]: Refusing update from remote peer 44d7da27cda843f7bc138c74ef574202: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:56:57.081036 2663 consensus_queue.cc:1035] T 970a01413d4e4d26846dec7cf500062e P 44d7da27cda843f7bc138c74ef574202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5506b5c58aa4191be250c176296a85f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:56:57.081172 2691 consensus_queue.cc:1035] T 970a01413d4e4d26846dec7cf500062e P 44d7da27cda843f7bc138c74ef574202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d19f76cccc1c4fe192f0503263707fae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 14:56:57.094874 2689 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 14:56:58.653870 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:56:58.654047 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:56:58.656541 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 132a9d51505746b4a773d526a0d7c474: Sending DeleteTablet for 3 replicas of tablet 970a01413d4e4d26846dec7cf500062e
I20260927 14:56:58.657155 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 970a01413d4e4d26846dec7cf500062e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:56:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32920
I20260927 14:56:58.657210 2527 tablet_service.cc:1514] Processing DeleteTablet for tablet 970a01413d4e4d26846dec7cf500062e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:56:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44202
I20260927 14:56:58.657208 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet 970a01413d4e4d26846dec7cf500062e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:56:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33084
I20260927 14:56:58.657575 2759 tablet_replica.cc:330] T 970a01413d4e4d26846dec7cf500062e P c5506b5c58aa4191be250c176296a85f: stopping tablet replica
I20260927 14:56:58.657577 2758 tablet_replica.cc:330] T 970a01413d4e4d26846dec7cf500062e P 44d7da27cda843f7bc138c74ef574202: stopping tablet replica
I20260927 14:56:58.657579 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 2195
I20260927 14:56:58.658424 2760 tablet_replica.cc:330] T 970a01413d4e4d26846dec7cf500062e P d19f76cccc1c4fe192f0503263707fae: stopping tablet replica
I20260927 14:56:58.658607 2759 raft_consensus.cc:2229] T 970a01413d4e4d26846dec7cf500062e P c5506b5c58aa4191be250c176296a85f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:56:58.658813 2760 raft_consensus.cc:2229] T 970a01413d4e4d26846dec7cf500062e P d19f76cccc1c4fe192f0503263707fae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:56:58.658998 2759 raft_consensus.cc:2258] T 970a01413d4e4d26846dec7cf500062e P c5506b5c58aa4191be250c176296a85f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:56:58.659150 2760 raft_consensus.cc:2258] T 970a01413d4e4d26846dec7cf500062e P d19f76cccc1c4fe192f0503263707fae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:56:58.659833 2760 ts_tablet_manager.cc:1905] T 970a01413d4e4d26846dec7cf500062e P d19f76cccc1c4fe192f0503263707fae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:56:58.659946 2759 ts_tablet_manager.cc:1905] T 970a01413d4e4d26846dec7cf500062e P c5506b5c58aa4191be250c176296a85f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:56:58.662305 2760 ts_tablet_manager.cc:1918] T 970a01413d4e4d26846dec7cf500062e P d19f76cccc1c4fe192f0503263707fae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 14:56:58.662420 2760 log.cc:1196] T 970a01413d4e4d26846dec7cf500062e P d19f76cccc1c4fe192f0503263707fae: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-2/wal/wals/970a01413d4e4d26846dec7cf500062e
I20260927 14:56:58.662634 2759 ts_tablet_manager.cc:1918] T 970a01413d4e4d26846dec7cf500062e P c5506b5c58aa4191be250c176296a85f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 14:56:58.662719 2760 ts_tablet_manager.cc:1939] T 970a01413d4e4d26846dec7cf500062e P d19f76cccc1c4fe192f0503263707fae: Deleting consensus metadata
I20260927 14:56:58.662738 2759 log.cc:1196] T 970a01413d4e4d26846dec7cf500062e P c5506b5c58aa4191be250c176296a85f: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790521004237705-392-0/minicluster-data/ts-1/wal/wals/970a01413d4e4d26846dec7cf500062e
I20260927 14:56:58.663025 2759 ts_tablet_manager.cc:1939] T 970a01413d4e4d26846dec7cf500062e P c5506b5c58aa4191be250c176296a85f: Deleting consensus metadata
I20260927 14:56:58.663552 2110 catalog_manager.cc:4841] TS d19f76cccc1c4fe192f0503263707fae (127.0.98.3:43043): tablet 970a01413d4e4d26846dec7cf500062e (table test-table [id=a09761737933415fadfe63776853fbe0]) successfully deleted
I20260927 14:56:58.663729 2109 catalog_manager.cc:4841] TS c5506b5c58aa4191be250c176296a85f (127.0.98.2:35587): tablet 970a01413d4e4d26846dec7cf500062e (table test-table [id=a09761737933415fadfe63776853fbe0]) successfully deleted
W20260927 14:56:58.666016 2109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42795 (error 108)
I20260927 14:56:58.666173 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 2335
I20260927 14:56:58.668429 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:58.667155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42795 (local address 127.0.0.1:32952)
0927 14:56:58.668259 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0927 14:56:58.668332 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1033,"thread_start_us":79,"threads_started":1}
W20260927 14:56:58.668545 2109 catalog_manager.cc:4584] TS 44d7da27cda843f7bc138c74ef574202 (127.0.98.1:42795): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 970a01413d4e4d26846dec7cf500062e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42795: connect: Connection refused (error 111)
I20260927 14:56:58.674256 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 2475
I20260927 14:56:58.680255 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 2091
2026-09-27T14:56:58Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6275 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:56:58 dist-test-slave-pz0h krb5kdc[2765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:56:58 dist-test-slave-pz0h krb5kdc[2765](info): set up 2 sockets
Sep 27 14:56:58 dist-test-slave-pz0h krb5kdc[2765](info): commencing operation
krb5kdc: starting...
W20260927 14:56:59.642529 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 14:56:59 dist-test-slave-pz0h krb5kdc[2765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:56:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:56:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:56:59.759748 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41591
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:56:59.875244 2778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:56:59.875551 2778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:56:59.875594 2778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:56:59.879760 2778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:56:59.879833 2778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:56:59.879853 2778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:56:59.879868 2778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:56:59.879881 2778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:56:59.885562 2778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:56:59.886651 2778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:56:59.887709 2778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:56:59.896266 2783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:56:59.896245 2784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:59.896296 2778 server_base.cc:1034] running on GCE node
W20260927 14:56:59.896337 2786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:56:59.897147 2778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:56:59.898375 2778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:56:59.899583 2778 hybrid_clock.cc:648] HybridClock initialized: now 1790521019899568 us; error 32 us; skew 500 ppm
Sep 27 14:56:59 dist-test-slave-pz0h krb5kdc[2765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:56:59.902475 2778 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:56:59.903461 2778 webserver.cc:460] Webserver started at http://127.0.98.62:40461/ using document root <none> and password file <none>
I20260927 14:56:59.904027 2778 fs_manager.cc:359] Metadata directory not provided
I20260927 14:56:59.904094 2778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:56:59.904289 2778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:56:59.906428 2778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "e2d6d8f137234853be0c38cb282b6beb"
format_stamp: "Formatted at 2026-09-27 14:56:59 on dist-test-slave-pz0h"
server_key: "09ff9ac4e3ef8aec69a55bf54afe78b6"
server_key_iv: "0b1650503de4836e3484ef934a029e79"
server_key_version: "encryptionkey@0"
I20260927 14:56:59.907024 2778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "e2d6d8f137234853be0c38cb282b6beb"
format_stamp: "Formatted at 2026-09-27 14:56:59 on dist-test-slave-pz0h"
server_key: "09ff9ac4e3ef8aec69a55bf54afe78b6"
server_key_iv: "0b1650503de4836e3484ef934a029e79"
server_key_version: "encryptionkey@0"
I20260927 14:56:59.910934 2778 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 14:56:59.913667 2793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:56:59.915072 2778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:56:59.915217 2778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "e2d6d8f137234853be0c38cb282b6beb"
format_stamp: "Formatted at 2026-09-27 14:56:59 on dist-test-slave-pz0h"
server_key: "09ff9ac4e3ef8aec69a55bf54afe78b6"
server_key_iv: "0b1650503de4836e3484ef934a029e79"
server_key_version: "encryptionkey@0"
I20260927 14:56:59.915336 2778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:56:59.933162 2778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:56:59.937191 2778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:56:59.937412 2778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:56:59.944655 2778 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41591
I20260927 14:56:59.944674 2845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41591 every 8 connection(s)
I20260927 14:56:59.945680 2778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:56:59.946705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 2778
I20260927 14:56:59.946841 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 14:56:59.947120 392 external_mini_cluster.cc:1428] Setting key 23d5b0eec9c5a0c6438f71df60d4529c
I20260927 14:56:59.949126 2846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:56:59.955936 2846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2d6d8f137234853be0c38cb282b6beb: Bootstrap starting.
Sep 27 14:56:59 dist-test-slave-pz0h krb5kdc[2765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:56:59.958354 2846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2d6d8f137234853be0c38cb282b6beb: Neither blocks nor log segments found. Creating new log.
I20260927 14:56:59.959146 2846 log.cc:824] T 00000000000000000000000000000000 P e2d6d8f137234853be0c38cb282b6beb: Log is configured to *not* fsync() on all Append() calls
I20260927 14:56:59.961588 2846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2d6d8f137234853be0c38cb282b6beb: No bootstrap required, opened a new log
I20260927 14:56:59.962551 2849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:56:59.948514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50520 (local address 127.0.98.62:41591)
0927 14:56:59.948953 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0927 14:56:59.948969 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 14:56:59.949009 (+ 40us) server_negotiation.cc:407] Connection header received
0927 14:56:59.949820 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:56:59.949841 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:56:59.950207 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:56:59.950542 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:56:59.951641 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:59.952720 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:59.953921 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:56:59.954431 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:56:59.954573 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:56:59.957432 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:56:59.957438 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:56:59.957453 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:56:59.957490 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:56:59.959258 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:59.959787 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:59.959793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:59.959797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:59.959922 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:56:59.960203 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:56:59.960206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:56:59.960208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:56:59.960729 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:56:59.960930 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 14:56:59.961198 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 14:56:59.961522 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":126,"threads_started":1}
I20260927 14:56:59.965431 2846 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2d6d8f137234853be0c38cb282b6beb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d6d8f137234853be0c38cb282b6beb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41591 } }
I20260927 14:56:59.965687 2846 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2d6d8f137234853be0c38cb282b6beb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:56:59.965721 2846 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2d6d8f137234853be0c38cb282b6beb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2d6d8f137234853be0c38cb282b6beb, State: Initialized, Role: FOLLOWER
I20260927 14:56:59.966121 2846 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2d6d8f137234853be0c38cb282b6beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d6d8f137234853be0c38cb282b6beb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41591 } }
I20260927 14:56:59.966238 2846 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2d6d8f137234853be0c38cb282b6beb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:56:59.966284 2846 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2d6d8f137234853be0c38cb282b6beb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:56:59.966359 2846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2d6d8f137234853be0c38cb282b6beb [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:56:59.967394 2846 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2d6d8f137234853be0c38cb282b6beb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d6d8f137234853be0c38cb282b6beb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41591 } }
I20260927 14:56:59.967733 2846 leader_election.cc:302] T 00000000000000000000000000000000 P e2d6d8f137234853be0c38cb282b6beb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2d6d8f137234853be0c38cb282b6beb; no voters:
I20260927 14:56:59.968027 2846 leader_election.cc:288] T 00000000000000000000000000000000 P e2d6d8f137234853be0c38cb282b6beb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:56:59.968190 2851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2d6d8f137234853be0c38cb282b6beb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:56:59.968487 2851 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2d6d8f137234853be0c38cb282b6beb [term 1 LEADER]: Becoming Leader. State: Replica: e2d6d8f137234853be0c38cb282b6beb, State: Running, Role: LEADER
I20260927 14:56:59.968864 2851 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2d6d8f137234853be0c38cb282b6beb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d6d8f137234853be0c38cb282b6beb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41591 } }
I20260927 14:56:59.969340 2846 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2d6d8f137234853be0c38cb282b6beb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:56:59.970496 2852 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2d6d8f137234853be0c38cb282b6beb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2d6d8f137234853be0c38cb282b6beb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d6d8f137234853be0c38cb282b6beb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41591 } } }
I20260927 14:56:59.970625 2852 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2d6d8f137234853be0c38cb282b6beb [sys.catalog]: This master's current role is: LEADER
I20260927 14:56:59.970956 2853 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2d6d8f137234853be0c38cb282b6beb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2d6d8f137234853be0c38cb282b6beb. Latest consensus state: current_term: 1 leader_uuid: "e2d6d8f137234853be0c38cb282b6beb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d6d8f137234853be0c38cb282b6beb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41591 } } }
I20260927 14:56:59.971071 2860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:56:59.971114 2853 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2d6d8f137234853be0c38cb282b6beb [sys.catalog]: This master's current role is: LEADER
I20260927 14:56:59.974030 2860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:56:59.980366 2860 catalog_manager.cc:1347] Generated new cluster ID: 7ad7f60b780d471d8725ce3604cdd50a
I20260927 14:56:59.980461 2860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:56:59.999938 2860 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:57:00.001313 2860 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:57:00.008074 2860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2d6d8f137234853be0c38cb282b6beb: Generated new TSK 0
I20260927 14:57:00.008831 2860 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:57:00.035872 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41591
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:00.149899 2872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:00.150202 2872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:00.150269 2872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:00.154428 2872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:00.154510 2872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:00.154593 2872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:57:00.160065 2872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:00.161183 2872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:00.162215 2872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:00.172034 2880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:00.172037 2878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:00.172096 2872 server_base.cc:1034] running on GCE node
W20260927 14:57:00.172037 2877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:00.172695 2872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:00.173363 2872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:00.174537 2872 hybrid_clock.cc:648] HybridClock initialized: now 1790521020174513 us; error 34 us; skew 500 ppm
Sep 27 14:57:00 dist-test-slave-pz0h krb5kdc[2765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:00.177352 2872 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:57:00.178296 2872 webserver.cc:460] Webserver started at http://127.0.98.1:38691/ using document root <none> and password file <none>
I20260927 14:57:00.178910 2872 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:00.178993 2872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:00.179198 2872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:00.181298 2872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/data/instance:
uuid: "ee89ee7fa68c4c7996114a09a3f50729"
format_stamp: "Formatted at 2026-09-27 14:57:00 on dist-test-slave-pz0h"
server_key: "37798d7f3bc51225a211cfff9615ae6c"
server_key_iv: "c0c4fd59f854f6777c62a40ce8fc6906"
server_key_version: "encryptionkey@0"
I20260927 14:57:00.181852 2872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ee89ee7fa68c4c7996114a09a3f50729"
format_stamp: "Formatted at 2026-09-27 14:57:00 on dist-test-slave-pz0h"
server_key: "37798d7f3bc51225a211cfff9615ae6c"
server_key_iv: "c0c4fd59f854f6777c62a40ce8fc6906"
server_key_version: "encryptionkey@0"
I20260927 14:57:00.185818 2872 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 14:57:00.188458 2887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:00.189909 2872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 14:57:00.190037 2872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "ee89ee7fa68c4c7996114a09a3f50729"
format_stamp: "Formatted at 2026-09-27 14:57:00 on dist-test-slave-pz0h"
server_key: "37798d7f3bc51225a211cfff9615ae6c"
server_key_iv: "c0c4fd59f854f6777c62a40ce8fc6906"
server_key_version: "encryptionkey@0"
I20260927 14:57:00.190150 2872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:00.210534 2872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:00.214381 2872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:00.214748 2872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:00.216233 2872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:00.217537 2872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:00.217612 2872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:00.217669 2872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:00.217698 2872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:00.226533 2872 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40399
I20260927 14:57:00.226557 3000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40399 every 8 connection(s)
I20260927 14:57:00.227525 2872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:57:00.232811 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 2872
I20260927 14:57:00.232926 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:57:00.233186 392 external_mini_cluster.cc:1428] Setting key 1d53a75511ef380f883be5d5bc3f8446
Sep 27 14:57:00 dist-test-slave-pz0h krb5kdc[2765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 14:57:00.242656 2849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:00.229539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56179 (local address 127.0.98.62:41591)
0927 14:57:00.229677 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:00.229680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:00.230479 (+ 799us) server_negotiation.cc:407] Connection header received
0927 14:57:00.231898 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:00.231902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:00.231961 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:00.232036 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:00.233490 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:00.234077 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:00.235797 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:00.236052 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:00.236167 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:00.238986 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:00.238989 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:00.238991 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:00.239019 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:00.240447 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:00.241031 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:00.241034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:00.241035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:00.241119 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:00.241476 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:00.241479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:00.241480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:00.241681 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:00.241775 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:00.242332 (+ 557us) server_negotiation.cc:299] Negotiation successful
0927 14:57:00.242478 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 14:57:00.243589 3003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:00.229856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41591 (local address 127.0.98.1:56179)
0927 14:57:00.230328 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:00.230372 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:00.231682 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:00.232214 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:00.232222 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:00.232637 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:00.233329 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:00.233350 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:00.234226 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:00.234229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:00.235636 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:00.235650 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:00.236222 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:00.236227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:00.236450 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:00.237042 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:00.237060 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:00.238760 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:00.240588 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:00.240595 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:00.240617 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:00.240914 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:00.241209 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:00.241213 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:00.241215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:00.241373 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:00.241783 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:00.241791 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:00.242066 (+ 275us) client_negotiation.cc:769] Sending connection context
0927 14:57:00.242254 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 14:57:00.242621 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":125,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 14:57:00.244975 3001 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41591
I20260927 14:57:00.245252 3001 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:00.245805 3001 heartbeater.cc:511] Master 127.0.98.62:41591 requested a full tablet report, sending...
I20260927 14:57:00.247619 2810 ts_manager.cc:194] Registered new tserver with Master: ee89ee7fa68c4c7996114a09a3f50729 (127.0.98.1:40399)
I20260927 14:57:00.248772 2810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:57:00.262388 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41591
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:00.373554 3006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:00.373844 3006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:00.373888 3006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:00.378051 3006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:00.378129 3006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:00.378191 3006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:57:00.383818 3006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:00.384905 3006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:00.385900 3006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:00.395660 3012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:00.395669 3014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:00.395673 3011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:00.395886 3006 server_base.cc:1034] running on GCE node
I20260927 14:57:00.396385 3006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:00.397089 3006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:00.398255 3006 hybrid_clock.cc:648] HybridClock initialized: now 1790521020398225 us; error 46 us; skew 500 ppm
Sep 27 14:57:00 dist-test-slave-pz0h krb5kdc[2765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:00.401127 3006 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:57:00.402122 3006 webserver.cc:460] Webserver started at http://127.0.98.2:35105/ using document root <none> and password file <none>
I20260927 14:57:00.402848 3006 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:00.402936 3006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:00.403155 3006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:00.405364 3006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/data/instance:
uuid: "60868ded524e4b6aaad8a0d8276ebb98"
format_stamp: "Formatted at 2026-09-27 14:57:00 on dist-test-slave-pz0h"
server_key: "afd136f801002db109f235aa0a194a46"
server_key_iv: "ae3012d077f84d4b0a150c3a1517f94d"
server_key_version: "encryptionkey@0"
I20260927 14:57:00.405966 3006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "60868ded524e4b6aaad8a0d8276ebb98"
format_stamp: "Formatted at 2026-09-27 14:57:00 on dist-test-slave-pz0h"
server_key: "afd136f801002db109f235aa0a194a46"
server_key_iv: "ae3012d077f84d4b0a150c3a1517f94d"
server_key_version: "encryptionkey@0"
I20260927 14:57:00.410043 3006 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 14:57:00.412968 3021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:00.414743 3006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 14:57:00.414920 3006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "60868ded524e4b6aaad8a0d8276ebb98"
format_stamp: "Formatted at 2026-09-27 14:57:00 on dist-test-slave-pz0h"
server_key: "afd136f801002db109f235aa0a194a46"
server_key_iv: "ae3012d077f84d4b0a150c3a1517f94d"
server_key_version: "encryptionkey@0"
I20260927 14:57:00.415047 3006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:00.440189 3006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:00.444765 3006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:00.444998 3006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:00.445637 3006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:00.446521 3006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:00.446568 3006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:00.446648 3006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:00.446674 3006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:00.457237 3006 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36115
I20260927 14:57:00.457262 3134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36115 every 8 connection(s)
I20260927 14:57:00.458268 3006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:57:00.459033 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 3006
I20260927 14:57:00.459146 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:57:00.459410 392 external_mini_cluster.cc:1428] Setting key 85fb1cd22b2a079b23d81f802033606c
Sep 27 14:57:00 dist-test-slave-pz0h krb5kdc[2765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 14:57:00.473795 2849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:00.460229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43067 (local address 127.0.98.62:41591)
0927 14:57:00.460347 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:00.460351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:00.461461 (+ 1110us) server_negotiation.cc:407] Connection header received
0927 14:57:00.463444 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:00.463447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:00.463501 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:00.463572 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:00.465062 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:00.465579 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:00.466858 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:00.467091 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:00.467183 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:00.470039 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:00.470043 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:00.470046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:00.470077 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:00.471614 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:00.472128 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:00.472130 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:00.472131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:00.472244 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:00.472623 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:00.472627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:00.472629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:00.472870 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:00.472980 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:00.473487 (+ 507us) server_negotiation.cc:299] Negotiation successful
0927 14:57:00.473624 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 14:57:00.474819 3137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:00.460758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41591 (local address 127.0.98.2:43067)
0927 14:57:00.461277 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:00.461327 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:00.463149 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:00.463801 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:00.463810 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:00.464346 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:00.464895 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:00.464911 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:00.465711 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:00.465714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:00.466700 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:00.466713 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:00.467247 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:00.467251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:00.467436 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:00.468059 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:00.468074 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:00.469827 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:00.471745 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:00.471750 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:00.471764 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:00.472045 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:00.472339 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:00.472343 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:00.472345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:00.472533 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:00.472982 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:00.472987 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:00.473258 (+ 271us) client_negotiation.cc:769] Sending connection context
0927 14:57:00.473438 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 14:57:00.473715 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":148,"threads_started":1}
I20260927 14:57:00.476032 3135 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41591
I20260927 14:57:00.476274 3135 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:00.476809 3135 heartbeater.cc:511] Master 127.0.98.62:41591 requested a full tablet report, sending...
I20260927 14:57:00.477960 2810 ts_manager.cc:194] Registered new tserver with Master: 60868ded524e4b6aaad8a0d8276ebb98 (127.0.98.2:36115)
I20260927 14:57:00.478597 2810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43067
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:57:00.487639 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41591
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:00.599591 3140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:00.599888 3140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:00.599944 3140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:00.604269 3140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:00.604343 3140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:00.604409 3140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:57:00.610029 3140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:00.611227 3140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:00.612290 3140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:00.622364 3146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:00.622396 3145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:00.622424 3148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:00.622560 3140 server_base.cc:1034] running on GCE node
I20260927 14:57:00.623188 3140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:00.623886 3140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:00.625074 3140 hybrid_clock.cc:648] HybridClock initialized: now 1790521020625042 us; error 46 us; skew 500 ppm
Sep 27 14:57:00 dist-test-slave-pz0h krb5kdc[2765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:00.628090 3140 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:57:00.629069 3140 webserver.cc:460] Webserver started at http://127.0.98.3:40867/ using document root <none> and password file <none>
I20260927 14:57:00.629626 3140 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:00.629690 3140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:00.629876 3140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:00.632057 3140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/data/instance:
uuid: "62b6e904a18a4de8acd421d941407bd6"
format_stamp: "Formatted at 2026-09-27 14:57:00 on dist-test-slave-pz0h"
server_key: "6f0faf626f5143df08d8ea7a6c90de43"
server_key_iv: "14c3df6603de92919d1d819cb37521bf"
server_key_version: "encryptionkey@0"
I20260927 14:57:00.632642 3140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "62b6e904a18a4de8acd421d941407bd6"
format_stamp: "Formatted at 2026-09-27 14:57:00 on dist-test-slave-pz0h"
server_key: "6f0faf626f5143df08d8ea7a6c90de43"
server_key_iv: "14c3df6603de92919d1d819cb37521bf"
server_key_version: "encryptionkey@0"
I20260927 14:57:00.636752 3140 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 14:57:00.639434 3155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:00.640697 3140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:57:00.640800 3140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "62b6e904a18a4de8acd421d941407bd6"
format_stamp: "Formatted at 2026-09-27 14:57:00 on dist-test-slave-pz0h"
server_key: "6f0faf626f5143df08d8ea7a6c90de43"
server_key_iv: "14c3df6603de92919d1d819cb37521bf"
server_key_version: "encryptionkey@0"
I20260927 14:57:00.640899 3140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:00.672577 3140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:00.675900 3140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:00.676067 3140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:00.676630 3140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:00.677459 3140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:00.677505 3140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:00.677546 3140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:00.677565 3140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:00.686228 3140 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44947
I20260927 14:57:00.686254 3268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44947 every 8 connection(s)
I20260927 14:57:00.687794 3140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:57:00.694231 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 3140
I20260927 14:57:00.694339 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:57:00.694626 392 external_mini_cluster.cc:1428] Setting key 45258548457b69f522f2c05046baf469
Sep 27 14:57:00 dist-test-slave-pz0h krb5kdc[2765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:57:00.702577 2849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:00.689874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37453 (local address 127.0.98.62:41591)
0927 14:57:00.690013 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:00.690018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:00.691094 (+ 1076us) server_negotiation.cc:407] Connection header received
0927 14:57:00.692496 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:00.692499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:00.692549 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:00.692627 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:00.693840 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:00.694400 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:00.696027 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:00.696247 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:00.696341 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:00.699035 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:00.699039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:00.699041 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:00.699069 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:00.700392 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:00.700980 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:00.700984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:00.700985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:00.701066 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:00.701407 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:00.701409 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:00.701411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:00.701603 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:00.701688 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:00.702275 (+ 587us) server_negotiation.cc:299] Negotiation successful
0927 14:57:00.702409 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 14:57:00.703737 3271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:00.690288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41591 (local address 127.0.98.3:37453)
0927 14:57:00.690925 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:00.690978 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:00.692278 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:00.692811 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:00.692819 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:00.693211 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:00.693710 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:00.693726 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:00.694554 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:00.694558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:00.695904 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:00.695918 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:00.696411 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:00.696416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:00.696609 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:00.697218 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:00.697236 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:00.698789 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:00.700526 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:00.700534 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:00.700551 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:00.700868 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:00.701153 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:00.701156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:00.701158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:00.701311 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:00.701702 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:00.701709 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:00.702017 (+ 308us) client_negotiation.cc:769] Sending connection context
0927 14:57:00.702193 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 14:57:00.702571 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":147,"threads_started":1}
I20260927 14:57:00.705670 3269 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41591
I20260927 14:57:00.706079 3269 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:00.706887 3269 heartbeater.cc:511] Master 127.0.98.62:41591 requested a full tablet report, sending...
I20260927 14:57:00.708503 2810 ts_manager.cc:194] Registered new tserver with Master: 62b6e904a18a4de8acd421d941407bd6 (127.0.98.3:44947)
I20260927 14:57:00.709335 2810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37453
I20260927 14:57:00.709700 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 14:57:00.723603 2849 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:00.721978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50530 (local address 127.0.98.62:41591)
0927 14:57:00.722108 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:00.722113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:00.722305 (+ 192us) server_negotiation.cc:407] Connection header received
0927 14:57:00.722457 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:00.722460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:00.722504 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:00.722580 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:00.723459 (+ 879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50530: BlockingRecv error: recv got EOF from 127.0.0.1:50530 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 14:57:00.724828 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 2872
I20260927 14:57:00.730401 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 3006
I20260927 14:57:00.735188 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 3140
I20260927 14:57:00.740181 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 2778
2026-09-27T14:57:00Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2059 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:57:00 dist-test-slave-pz0h krb5kdc[3281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:57:00 dist-test-slave-pz0h krb5kdc[3281](info): set up 2 sockets
Sep 27 14:57:00 dist-test-slave-pz0h krb5kdc[3281](info): commencing operation
krb5kdc: starting...
W20260927 14:57:01.698587 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 14:57:01 dist-test-slave-pz0h krb5kdc[3281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:57:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:57:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:57:01.819202 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42195
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:01.936041 3294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:01.936349 3294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:01.936394 3294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:01.940559 3294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:57:01.940644 3294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:01.940662 3294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:01.940680 3294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:57:01.940694 3294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:57:01.946348 3294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:01.947490 3294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:01.948485 3294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:01.957011 3300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:01.957002 3302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:01.957002 3299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:01.957415 3294 server_base.cc:1034] running on GCE node
I20260927 14:57:01.958052 3294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:01.959327 3294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:01.960551 3294 hybrid_clock.cc:648] HybridClock initialized: now 1790521021960525 us; error 47 us; skew 500 ppm
Sep 27 14:57:01 dist-test-slave-pz0h krb5kdc[3281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521021, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:01.970759 3294 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:57:01.971956 3294 webserver.cc:460] Webserver started at http://127.0.98.62:38617/ using document root <none> and password file <none>
I20260927 14:57:01.972550 3294 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:01.972625 3294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:01.972831 3294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:01.975917 3294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "ad44008addd44fd1bd66aa0c0fa806c1"
format_stamp: "Formatted at 2026-09-27 14:57:01 on dist-test-slave-pz0h"
server_key: "8a0ec31bef4c55b4f026057c17e62cbd"
server_key_iv: "fedcb88d77b40827be781f3662b5c989"
server_key_version: "encryptionkey@0"
I20260927 14:57:01.976600 3294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "ad44008addd44fd1bd66aa0c0fa806c1"
format_stamp: "Formatted at 2026-09-27 14:57:01 on dist-test-slave-pz0h"
server_key: "8a0ec31bef4c55b4f026057c17e62cbd"
server_key_iv: "fedcb88d77b40827be781f3662b5c989"
server_key_version: "encryptionkey@0"
I20260927 14:57:01.981757 3294 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 14:57:01.985037 3309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:01.986584 3294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:57:01.986737 3294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "ad44008addd44fd1bd66aa0c0fa806c1"
format_stamp: "Formatted at 2026-09-27 14:57:01 on dist-test-slave-pz0h"
server_key: "8a0ec31bef4c55b4f026057c17e62cbd"
server_key_iv: "fedcb88d77b40827be781f3662b5c989"
server_key_version: "encryptionkey@0"
I20260927 14:57:01.986909 3294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:01.999531 3294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:02.003108 3294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:02.003300 3294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:02.010279 3294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42195
I20260927 14:57:02.010551 3361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42195 every 8 connection(s)
I20260927 14:57:02.011340 3294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:57:02.014129 3362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:02.015887 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 3294
I20260927 14:57:02.015982 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 14:57:02.016214 392 external_mini_cluster.cc:1428] Setting key a024e931c5667f9eda0c2f563dcc0697
I20260927 14:57:02.020895 3362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad44008addd44fd1bd66aa0c0fa806c1: Bootstrap starting.
I20260927 14:57:02.023700 3362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad44008addd44fd1bd66aa0c0fa806c1: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:02.024574 3362 log.cc:824] T 00000000000000000000000000000000 P ad44008addd44fd1bd66aa0c0fa806c1: Log is configured to *not* fsync() on all Append() calls
Sep 27 14:57:02 dist-test-slave-pz0h krb5kdc[3281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:57:02.027104 3362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad44008addd44fd1bd66aa0c0fa806c1: No bootstrap required, opened a new log
I20260927 14:57:02.031862 3362 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad44008addd44fd1bd66aa0c0fa806c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad44008addd44fd1bd66aa0c0fa806c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42195 } }
I20260927 14:57:02.032188 3362 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad44008addd44fd1bd66aa0c0fa806c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:02.032239 3362 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad44008addd44fd1bd66aa0c0fa806c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad44008addd44fd1bd66aa0c0fa806c1, State: Initialized, Role: FOLLOWER
I20260927 14:57:02.032603 3365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:02.017525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32920 (local address 127.0.98.62:42195)
0927 14:57:02.017969 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:02.017988 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:02.018031 (+ 43us) server_negotiation.cc:407] Connection header received
0927 14:57:02.018912 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:02.018938 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:02.019389 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:02.019793 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:02.020479 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:02.021923 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:02.022840 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:02.023525 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:02.023692 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:02.026458 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:02.026465 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:02.026483 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:02.026517 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:02.028543 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:02.029100 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:02.029108 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:02.029116 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:02.029303 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:02.029646 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:02.029649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:02.029651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:02.030247 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:02.030456 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:02.031088 (+ 632us) server_negotiation.cc:299] Negotiation successful
0927 14:57:02.031485 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":111,"threads_started":1}
I20260927 14:57:02.032774 3362 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad44008addd44fd1bd66aa0c0fa806c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad44008addd44fd1bd66aa0c0fa806c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42195 } }
I20260927 14:57:02.032981 3362 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad44008addd44fd1bd66aa0c0fa806c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:57:02.033041 3362 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad44008addd44fd1bd66aa0c0fa806c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:57:02.033156 3362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad44008addd44fd1bd66aa0c0fa806c1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:02.034356 3362 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad44008addd44fd1bd66aa0c0fa806c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad44008addd44fd1bd66aa0c0fa806c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42195 } }
I20260927 14:57:02.034742 3362 leader_election.cc:302] T 00000000000000000000000000000000 P ad44008addd44fd1bd66aa0c0fa806c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad44008addd44fd1bd66aa0c0fa806c1; no voters:
I20260927 14:57:02.035156 3362 leader_election.cc:288] T 00000000000000000000000000000000 P ad44008addd44fd1bd66aa0c0fa806c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:57:02.035403 3367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad44008addd44fd1bd66aa0c0fa806c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:57:02.035718 3367 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad44008addd44fd1bd66aa0c0fa806c1 [term 1 LEADER]: Becoming Leader. State: Replica: ad44008addd44fd1bd66aa0c0fa806c1, State: Running, Role: LEADER
I20260927 14:57:02.036154 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:57:02.036125 3367 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad44008addd44fd1bd66aa0c0fa806c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad44008addd44fd1bd66aa0c0fa806c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42195 } }
I20260927 14:57:02.036520 3362 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad44008addd44fd1bd66aa0c0fa806c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:57:02.037899 3368 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad44008addd44fd1bd66aa0c0fa806c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad44008addd44fd1bd66aa0c0fa806c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad44008addd44fd1bd66aa0c0fa806c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42195 } } }
I20260927 14:57:02.038036 3368 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad44008addd44fd1bd66aa0c0fa806c1 [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:02.038436 3382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:57:02.038442 3369 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad44008addd44fd1bd66aa0c0fa806c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad44008addd44fd1bd66aa0c0fa806c1. Latest consensus state: current_term: 1 leader_uuid: "ad44008addd44fd1bd66aa0c0fa806c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad44008addd44fd1bd66aa0c0fa806c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42195 } } }
I20260927 14:57:02.038942 3369 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad44008addd44fd1bd66aa0c0fa806c1 [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:02.042366 3382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:57:02.048624 3382 catalog_manager.cc:1347] Generated new cluster ID: 1b46df0318a644819ad8bff124ee4cc5
I20260927 14:57:02.048705 3382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:57:02.055536 3382 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:57:02.057137 3365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:02.051853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32924 (local address 127.0.98.62:42195)
0927 14:57:02.051977 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:02.051982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:02.052290 (+ 308us) server_negotiation.cc:407] Connection header received
0927 14:57:02.052436 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:02.052446 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:02.052494 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:02.052590 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:02.053213 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:02.053897 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:02.054787 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:02.055183 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:02.055293 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:02.056208 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:02.056212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:02.056214 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:57:02.056694 (+ 480us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:02.056727 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:57:02.056744 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:02.056823 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:02.056918 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 14:57:02.056985 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":43}
I20260927 14:57:02.057154 3382 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:57:02.069907 3382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad44008addd44fd1bd66aa0c0fa806c1: Generated new TSK 0
I20260927 14:57:02.070897 3382 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:57:02.084810 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 3294
2026-09-27T14:57:02Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1342 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:57:02 dist-test-slave-pz0h krb5kdc[3395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:57:02 dist-test-slave-pz0h krb5kdc[3395](info): set up 2 sockets
Sep 27 14:57:02 dist-test-slave-pz0h krb5kdc[3395](info): commencing operation
krb5kdc: starting...
W20260927 14:57:03.034665 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 14:57:03 dist-test-slave-pz0h krb5kdc[3395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:57:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:57:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:57:03.153448 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33479
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:03.267620 3408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:03.267931 3408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:03.267976 3408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:03.272172 3408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:57:03.272254 3408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:03.272274 3408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:03.272290 3408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:57:03.272303 3408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:57:03.277858 3408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:03.279104 3408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:03.280222 3408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:03.288894 3414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:03.288894 3413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:03.288897 3416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:03.289219 3408 server_base.cc:1034] running on GCE node
I20260927 14:57:03.289951 3408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:03.291234 3408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:03.292434 3408 hybrid_clock.cc:648] HybridClock initialized: now 1790521023292418 us; error 34 us; skew 500 ppm
Sep 27 14:57:03 dist-test-slave-pz0h krb5kdc[3395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:03.295420 3408 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:57:03.296371 3408 webserver.cc:460] Webserver started at http://127.0.98.62:46039/ using document root <none> and password file <none>
I20260927 14:57:03.296911 3408 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:03.296974 3408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:03.297149 3408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:03.299335 3408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "ee94fc04bb8441d182944423072b8802"
format_stamp: "Formatted at 2026-09-27 14:57:03 on dist-test-slave-pz0h"
server_key: "82773aad0db2f48cdf6e7629bb5f43be"
server_key_iv: "38618aeba34e465a52aa6755bb199ad5"
server_key_version: "encryptionkey@0"
I20260927 14:57:03.299916 3408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "ee94fc04bb8441d182944423072b8802"
format_stamp: "Formatted at 2026-09-27 14:57:03 on dist-test-slave-pz0h"
server_key: "82773aad0db2f48cdf6e7629bb5f43be"
server_key_iv: "38618aeba34e465a52aa6755bb199ad5"
server_key_version: "encryptionkey@0"
I20260927 14:57:03.303972 3408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 14:57:03.306674 3423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:03.308099 3408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:57:03.308247 3408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "ee94fc04bb8441d182944423072b8802"
format_stamp: "Formatted at 2026-09-27 14:57:03 on dist-test-slave-pz0h"
server_key: "82773aad0db2f48cdf6e7629bb5f43be"
server_key_iv: "38618aeba34e465a52aa6755bb199ad5"
server_key_version: "encryptionkey@0"
I20260927 14:57:03.308382 3408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:03.330093 3408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:03.333280 3408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:03.333454 3408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:03.340644 3408 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33479
I20260927 14:57:03.341032 3475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33479 every 8 connection(s)
I20260927 14:57:03.342036 3408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:57:03.346084 3476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:03.350535 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 3408
I20260927 14:57:03.350649 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 14:57:03.350941 392 external_mini_cluster.cc:1428] Setting key a85d10872798dea6f5445c0391756994
I20260927 14:57:03.353299 3476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee94fc04bb8441d182944423072b8802: Bootstrap starting.
I20260927 14:57:03.355979 3476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee94fc04bb8441d182944423072b8802: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:03.356853 3476 log.cc:824] T 00000000000000000000000000000000 P ee94fc04bb8441d182944423072b8802: Log is configured to *not* fsync() on all Append() calls
I20260927 14:57:03.359283 3476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee94fc04bb8441d182944423072b8802: No bootstrap required, opened a new log
Sep 27 14:57:03 dist-test-slave-pz0h krb5kdc[3395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:57:03.362988 3476 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee94fc04bb8441d182944423072b8802 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee94fc04bb8441d182944423072b8802" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33479 } }
I20260927 14:57:03.363260 3476 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee94fc04bb8441d182944423072b8802 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:03.363296 3476 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee94fc04bb8441d182944423072b8802 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee94fc04bb8441d182944423072b8802, State: Initialized, Role: FOLLOWER
I20260927 14:57:03.363807 3476 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee94fc04bb8441d182944423072b8802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee94fc04bb8441d182944423072b8802" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33479 } }
I20260927 14:57:03.363979 3476 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee94fc04bb8441d182944423072b8802 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:57:03.364053 3476 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee94fc04bb8441d182944423072b8802 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:57:03.364157 3476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee94fc04bb8441d182944423072b8802 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:03.365242 3476 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee94fc04bb8441d182944423072b8802 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee94fc04bb8441d182944423072b8802" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33479 } }
I20260927 14:57:03.365664 3476 leader_election.cc:302] T 00000000000000000000000000000000 P ee94fc04bb8441d182944423072b8802 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee94fc04bb8441d182944423072b8802; no voters:
I20260927 14:57:03.365985 3476 leader_election.cc:288] T 00000000000000000000000000000000 P ee94fc04bb8441d182944423072b8802 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:57:03.366137 3481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee94fc04bb8441d182944423072b8802 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:57:03.366442 3481 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee94fc04bb8441d182944423072b8802 [term 1 LEADER]: Becoming Leader. State: Replica: ee94fc04bb8441d182944423072b8802, State: Running, Role: LEADER
I20260927 14:57:03.366894 3479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:03.352407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48444 (local address 127.0.98.62:33479)
0927 14:57:03.352893 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:03.352909 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:03.352948 (+ 39us) server_negotiation.cc:407] Connection header received
0927 14:57:03.353634 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:03.353654 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:03.354035 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:03.354347 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:03.355290 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:03.356358 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:03.357486 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:03.358065 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:03.358192 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:03.361028 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:03.361035 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:03.361051 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:03.361082 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:03.363530 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:03.364084 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:03.364091 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:03.364097 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:03.364247 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:03.364520 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:03.364522 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:03.364524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:03.364956 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:03.365110 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:03.365440 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 14:57:03.365770 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":147,"threads_started":1}
I20260927 14:57:03.366849 3481 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee94fc04bb8441d182944423072b8802 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee94fc04bb8441d182944423072b8802" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33479 } }
I20260927 14:57:03.367296 3476 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee94fc04bb8441d182944423072b8802 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:57:03.368530 3482 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee94fc04bb8441d182944423072b8802 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee94fc04bb8441d182944423072b8802" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee94fc04bb8441d182944423072b8802" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33479 } } }
I20260927 14:57:03.368633 3482 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee94fc04bb8441d182944423072b8802 [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:03.368884 3483 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee94fc04bb8441d182944423072b8802 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee94fc04bb8441d182944423072b8802. Latest consensus state: current_term: 1 leader_uuid: "ee94fc04bb8441d182944423072b8802" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee94fc04bb8441d182944423072b8802" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33479 } } }
I20260927 14:57:03.368975 3483 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee94fc04bb8441d182944423072b8802 [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:03.369305 3491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:57:03.372506 3491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:57:03.378280 3491 catalog_manager.cc:1347] Generated new cluster ID: 1f76da239d174f7e99173da0df1e93a3
I20260927 14:57:03.378456 3491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:57:03.385344 3491 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:57:03.386343 3491 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:57:03.400483 3491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee94fc04bb8441d182944423072b8802: Generated new TSK 0
I20260927 14:57:03.401254 3491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:57:03.431597 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33479
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:03.548518 3502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:03.548817 3502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:03.548861 3502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:03.552963 3502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:03.553037 3502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:03.553099 3502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:57:03.558611 3502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:03.559733 3502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:03.560750 3502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 14:57:03.570875 3502 server_base.cc:1034] running on GCE node
W20260927 14:57:03.570832 3508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:03.570832 3507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:03.570832 3510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:03.571502 3502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:03.572214 3502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:03.573410 3502 hybrid_clock.cc:648] HybridClock initialized: now 1790521023573394 us; error 42 us; skew 500 ppm
Sep 27 14:57:03 dist-test-slave-pz0h krb5kdc[3395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:03.576784 3502 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:57:03.577812 3502 webserver.cc:460] Webserver started at http://127.0.98.1:36943/ using document root <none> and password file <none>
I20260927 14:57:03.578408 3502 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:03.578490 3502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:03.578703 3502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:03.580932 3502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/data/instance:
uuid: "534ab7877a89484b9314238478a6d5cf"
format_stamp: "Formatted at 2026-09-27 14:57:03 on dist-test-slave-pz0h"
server_key: "0dddfa9ad592058eae400fa96a07e08c"
server_key_iv: "293e38079bf2ab7f83ead4121ad77fbd"
server_key_version: "encryptionkey@0"
I20260927 14:57:03.581542 3502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "534ab7877a89484b9314238478a6d5cf"
format_stamp: "Formatted at 2026-09-27 14:57:03 on dist-test-slave-pz0h"
server_key: "0dddfa9ad592058eae400fa96a07e08c"
server_key_iv: "293e38079bf2ab7f83ead4121ad77fbd"
server_key_version: "encryptionkey@0"
I20260927 14:57:03.585582 3502 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 14:57:03.588272 3517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:03.589582 3502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 14:57:03.589710 3502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "534ab7877a89484b9314238478a6d5cf"
format_stamp: "Formatted at 2026-09-27 14:57:03 on dist-test-slave-pz0h"
server_key: "0dddfa9ad592058eae400fa96a07e08c"
server_key_iv: "293e38079bf2ab7f83ead4121ad77fbd"
server_key_version: "encryptionkey@0"
I20260927 14:57:03.589830 3502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:03.607918 3502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:03.611707 3502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:03.611956 3502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:03.612579 3502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:03.613442 3502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:03.613490 3502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:03.613533 3502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:03.613554 3502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:03.622489 3502 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35133
I20260927 14:57:03.622512 3630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35133 every 8 connection(s)
I20260927 14:57:03.623646 3502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:57:03.628182 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 3502
I20260927 14:57:03.628312 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:57:03.628629 392 external_mini_cluster.cc:1428] Setting key 27f7d0b0ffb82fa4846a2583402dcaa6
Sep 27 14:57:03 dist-test-slave-pz0h krb5kdc[3395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 14:57:03.639420 3479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:03.625601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53603 (local address 127.0.98.62:33479)
0927 14:57:03.625745 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:03.625750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:03.626816 (+ 1066us) server_negotiation.cc:407] Connection header received
0927 14:57:03.628309 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:03.628313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:03.628379 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:03.628485 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:03.629783 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:03.630478 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:03.631933 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:03.632162 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:03.632254 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:03.635677 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:03.635680 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:03.635682 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:03.635710 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:03.637101 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:03.637698 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:03.637701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:03.637703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:03.637787 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:03.638172 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:03.638174 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:03.638175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:03.638373 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:03.638461 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:03.639108 (+ 647us) server_negotiation.cc:299] Negotiation successful
0927 14:57:03.639248 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 14:57:03.640385 3633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:03.626051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33479 (local address 127.0.98.1:53603)
0927 14:57:03.626619 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:03.626666 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:03.628042 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:03.628688 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:03.628703 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:03.629116 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:03.629655 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:03.629671 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:03.630635 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:03.630639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:03.631771 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:03.631784 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:03.632332 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:03.632336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:03.632576 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:03.633352 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:03.633375 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:03.635456 (+ 2081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:03.637242 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:03.637250 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:03.637273 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:03.637579 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:03.637881 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:03.637884 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:03.637886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:03.638069 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:03.638480 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:03.638489 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:03.638827 (+ 338us) client_negotiation.cc:769] Sending connection context
0927 14:57:03.639034 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 14:57:03.639406 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":137,"threads_started":1}
I20260927 14:57:03.641710 3631 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33479
I20260927 14:57:03.641960 3631 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:03.642493 3631 heartbeater.cc:511] Master 127.0.98.62:33479 requested a full tablet report, sending...
I20260927 14:57:03.644091 3440 ts_manager.cc:194] Registered new tserver with Master: 534ab7877a89484b9314238478a6d5cf (127.0.98.1:35133)
I20260927 14:57:03.645253 3440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:57:03.653781 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33479
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:03.765839 3636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:03.766148 3636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:03.766208 3636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:03.770627 3636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:03.770704 3636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:03.770789 3636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:57:03.776296 3636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:03.777433 3636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:03.778492 3636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:03.788275 3642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:03.788297 3641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:03.788278 3644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:03.788622 3636 server_base.cc:1034] running on GCE node
I20260927 14:57:03.789052 3636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:03.789717 3636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:03.790867 3636 hybrid_clock.cc:648] HybridClock initialized: now 1790521023790848 us; error 31 us; skew 500 ppm
Sep 27 14:57:03 dist-test-slave-pz0h krb5kdc[3395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:03.793653 3636 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:57:03.794613 3636 webserver.cc:460] Webserver started at http://127.0.98.2:44845/ using document root <none> and password file <none>
I20260927 14:57:03.795199 3636 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:03.795306 3636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:03.795516 3636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:03.797559 3636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/data/instance:
uuid: "ce53863c85c34942940ba61f8171e791"
format_stamp: "Formatted at 2026-09-27 14:57:03 on dist-test-slave-pz0h"
server_key: "a1d5968fc7567c977ea271527a1afc93"
server_key_iv: "af79bd6978653e001b1aa8def793317e"
server_key_version: "encryptionkey@0"
I20260927 14:57:03.798076 3636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ce53863c85c34942940ba61f8171e791"
format_stamp: "Formatted at 2026-09-27 14:57:03 on dist-test-slave-pz0h"
server_key: "a1d5968fc7567c977ea271527a1afc93"
server_key_iv: "af79bd6978653e001b1aa8def793317e"
server_key_version: "encryptionkey@0"
I20260927 14:57:03.802093 3636 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 14:57:03.805178 3651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:03.806887 3636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:57:03.807037 3636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "ce53863c85c34942940ba61f8171e791"
format_stamp: "Formatted at 2026-09-27 14:57:03 on dist-test-slave-pz0h"
server_key: "a1d5968fc7567c977ea271527a1afc93"
server_key_iv: "af79bd6978653e001b1aa8def793317e"
server_key_version: "encryptionkey@0"
I20260927 14:57:03.807192 3636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:03.822892 3636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:03.826007 3636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:03.826191 3636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:03.826821 3636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:03.827723 3636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:03.827769 3636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:03.827811 3636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:03.827837 3636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:03.836213 3636 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39899
I20260927 14:57:03.836238 3764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39899 every 8 connection(s)
I20260927 14:57:03.837147 3636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:57:03.840318 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 3636
I20260927 14:57:03.840448 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:57:03.840699 392 external_mini_cluster.cc:1428] Setting key 8bffbca5ed7c56bd54885b785030d6b9
Sep 27 14:57:03 dist-test-slave-pz0h krb5kdc[3395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 14:57:03.852530 3479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:03.839079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36039 (local address 127.0.98.62:33479)
0927 14:57:03.839217 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:03.839221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:03.840152 (+ 931us) server_negotiation.cc:407] Connection header received
0927 14:57:03.841642 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:03.841646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:03.841700 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:03.841773 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:03.843141 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:03.843670 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:03.845006 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:03.845232 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:03.845329 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:03.848285 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:03.848288 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:03.848290 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:03.848381 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:03.849960 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:03.850649 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:03.850653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:03.850654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:03.850755 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:03.851225 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:03.851229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:03.851231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:03.851482 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:03.851579 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:03.852187 (+ 608us) server_negotiation.cc:299] Negotiation successful
0927 14:57:03.852336 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 14:57:03.853591 3767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:03.839489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33479 (local address 127.0.98.2:36039)
0927 14:57:03.839974 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:03.840040 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:03.841436 (+ 1396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:03.841975 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:03.841983 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:03.842408 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:03.842971 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:03.842987 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:03.843803 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:03.843806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:03.844855 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:03.844866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:03.845399 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:03.845403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:03.845583 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:03.846170 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:03.846187 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:03.848047 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:03.850146 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:03.850153 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:03.850176 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:03.850526 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:03.850884 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:03.850887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:03.850889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:03.851117 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:03.851592 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:03.851602 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:03.851917 (+ 315us) client_negotiation.cc:769] Sending connection context
0927 14:57:03.852123 (+ 206us) client_negotiation.cc:240] Negotiation successful
0927 14:57:03.852505 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":128,"threads_started":1}
I20260927 14:57:03.855055 3765 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33479
I20260927 14:57:03.855479 3765 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:03.856424 3765 heartbeater.cc:511] Master 127.0.98.62:33479 requested a full tablet report, sending...
I20260927 14:57:03.858269 3440 ts_manager.cc:194] Registered new tserver with Master: ce53863c85c34942940ba61f8171e791 (127.0.98.2:39899)
I20260927 14:57:03.859098 3440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:57:03.867307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33479
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:03.983302 3770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:03.983610 3770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:03.983656 3770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:03.988171 3770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:03.988255 3770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:03.988318 3770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:57:03.993968 3770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:03.995157 3770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:03.996227 3770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:04.007267 3778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:04.007347 3775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:04.007296 3776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:04.007712 3770 server_base.cc:1034] running on GCE node
I20260927 14:57:04.008185 3770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:04.008924 3770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:04.010098 3770 hybrid_clock.cc:648] HybridClock initialized: now 1790521024010076 us; error 36 us; skew 500 ppm
Sep 27 14:57:04 dist-test-slave-pz0h krb5kdc[3395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:04.013388 3770 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:57:04.014673 3770 webserver.cc:460] Webserver started at http://127.0.98.3:39103/ using document root <none> and password file <none>
I20260927 14:57:04.015323 3770 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:04.015401 3770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:04.015612 3770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:04.017818 3770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/data/instance:
uuid: "25b4002bf9d34908bf7d43583c23dd1d"
format_stamp: "Formatted at 2026-09-27 14:57:04 on dist-test-slave-pz0h"
server_key: "eb6c51eff57abaae54c7e28d2b281580"
server_key_iv: "0d51934a59262ab5ac37d0a519780bb1"
server_key_version: "encryptionkey@0"
I20260927 14:57:04.018400 3770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "25b4002bf9d34908bf7d43583c23dd1d"
format_stamp: "Formatted at 2026-09-27 14:57:04 on dist-test-slave-pz0h"
server_key: "eb6c51eff57abaae54c7e28d2b281580"
server_key_iv: "0d51934a59262ab5ac37d0a519780bb1"
server_key_version: "encryptionkey@0"
I20260927 14:57:04.022493 3770 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 14:57:04.025439 3785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:04.026737 3770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 14:57:04.026908 3770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "25b4002bf9d34908bf7d43583c23dd1d"
format_stamp: "Formatted at 2026-09-27 14:57:04 on dist-test-slave-pz0h"
server_key: "eb6c51eff57abaae54c7e28d2b281580"
server_key_iv: "0d51934a59262ab5ac37d0a519780bb1"
server_key_version: "encryptionkey@0"
I20260927 14:57:04.027027 3770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:04.059697 3770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:04.062885 3770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:04.063067 3770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:04.063671 3770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:04.064496 3770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:04.064543 3770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:04.064594 3770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:04.064622 3770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:04.073257 3770 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37819
I20260927 14:57:04.073284 3898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37819 every 8 connection(s)
I20260927 14:57:04.074170 3770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:57:04.074514 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 3770
I20260927 14:57:04.074623 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790521004237705-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:57:04.074967 392 external_mini_cluster.cc:1428] Setting key c1467bc5df5090847eedc8a701023faa
Sep 27 14:57:04 dist-test-slave-pz0h krb5kdc[3395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:57:04.088225 3479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:04.076129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37043 (local address 127.0.98.62:33479)
0927 14:57:04.076276 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:04.076281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:04.077305 (+ 1024us) server_negotiation.cc:407] Connection header received
0927 14:57:04.078691 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:04.078694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:04.078751 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:04.078869 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:04.080069 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:04.080594 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:04.081755 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:04.081975 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:04.082063 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:04.084626 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:04.084629 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:04.084631 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:04.084659 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:04.086044 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:04.086576 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:04.086579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:04.086581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:04.086659 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:04.087025 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:04.087028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:04.087030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:04.087215 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:04.087300 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:04.087934 (+ 634us) server_negotiation.cc:299] Negotiation successful
0927 14:57:04.088067 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:57:04.089185 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:04.076619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33479 (local address 127.0.98.3:37043)
0927 14:57:04.077166 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:04.077215 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:04.078479 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:04.079034 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:04.079042 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:04.079430 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:04.079927 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:04.079941 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:04.080715 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:04.080719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:04.081634 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:04.081643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:04.082121 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:04.082125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:04.082301 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:04.082912 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:04.082928 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:04.084426 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:04.086178 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:04.086185 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:04.086204 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:04.086479 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:04.086743 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:04.086746 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:04.086748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:04.086938 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:04.087312 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:04.087319 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:04.087647 (+ 328us) client_negotiation.cc:769] Sending connection context
0927 14:57:04.087861 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 14:57:04.088198 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":150,"threads_started":1}
I20260927 14:57:04.090144 3899 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33479
I20260927 14:57:04.090381 3899 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:04.090904 3899 heartbeater.cc:511] Master 127.0.98.62:33479 requested a full tablet report, sending...
I20260927 14:57:04.091867 3440 ts_manager.cc:194] Registered new tserver with Master: 25b4002bf9d34908bf7d43583c23dd1d (127.0.98.3:37819)
I20260927 14:57:04.092408 3440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37043
I20260927 14:57:04.100749 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 14:57:04 dist-test-slave-pz0h krb5kdc[3395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521024, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 14:57:04 dist-test-slave-pz0h krb5kdc[3395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521024, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:57:04.125348 3479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:04.117324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48456 (local address 127.0.98.62:33479)
0927 14:57:04.117490 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:04.117495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:04.117625 (+ 130us) server_negotiation.cc:407] Connection header received
0927 14:57:04.117718 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:04.117721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:04.117770 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:04.117847 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:04.118472 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:04.119052 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:04.119904 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:04.120131 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:04.120220 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:04.122637 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:04.122641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:04.122643 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:04.122671 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:04.123980 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:04.124323 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:04.124325 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:04.124326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:04.124392 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:04.124576 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:04.124578 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:04.124580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:04.124784 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:04.124885 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:04.125085 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 14:57:04.125176 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 14:57:04.125737 3440 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48456
I20260927 14:57:04.126626 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 3502
I20260927 14:57:04.131520 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 3636
I20260927 14:57:04.136127 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 3770
I20260927 14:57:04.140873 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 3408
2026-09-27T14:57:04Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2057 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:57:04 dist-test-slave-pz0h krb5kdc[3911](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:57:04 dist-test-slave-pz0h krb5kdc[3911](info): set up 2 sockets
Sep 27 14:57:04 dist-test-slave-pz0h krb5kdc[3911](info): commencing operation
krb5kdc: starting...
W20260927 14:57:05.094642 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 14:57:05 dist-test-slave-pz0h krb5kdc[3911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:57:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:57:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:57:05.212090 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40529
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:05.326697 3924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:05.327069 3924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:05.327114 3924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:05.331338 3924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:57:05.331429 3924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:05.331447 3924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:05.331463 3924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:57:05.331477 3924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:57:05.337072 3924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:05.338222 3924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:05.339283 3924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:05.348105 3930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:05.348104 3929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:05.348467 3924 server_base.cc:1034] running on GCE node
W20260927 14:57:05.348109 3932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:05.349218 3924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:05.350494 3924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:05.351703 3924 hybrid_clock.cc:648] HybridClock initialized: now 1790521025351668 us; error 55 us; skew 500 ppm
Sep 27 14:57:05 dist-test-slave-pz0h krb5kdc[3911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:05.354915 3924 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:57:05.355995 3924 webserver.cc:460] Webserver started at http://127.0.98.62:35901/ using document root <none> and password file <none>
I20260927 14:57:05.356580 3924 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:05.356650 3924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:05.356853 3924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:05.359122 3924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "5962d34d84d74960ab0fa53f2c4e4a7f"
format_stamp: "Formatted at 2026-09-27 14:57:05 on dist-test-slave-pz0h"
server_key: "30e9bb62034cb96c89afc201e91d8c5d"
server_key_iv: "88edbfdef408b5c3717b10d2bc52bb0c"
server_key_version: "encryptionkey@0"
I20260927 14:57:05.359714 3924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "5962d34d84d74960ab0fa53f2c4e4a7f"
format_stamp: "Formatted at 2026-09-27 14:57:05 on dist-test-slave-pz0h"
server_key: "30e9bb62034cb96c89afc201e91d8c5d"
server_key_iv: "88edbfdef408b5c3717b10d2bc52bb0c"
server_key_version: "encryptionkey@0"
I20260927 14:57:05.364023 3924 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 14:57:05.366739 3939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:05.368187 3924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 14:57:05.368336 3924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "5962d34d84d74960ab0fa53f2c4e4a7f"
format_stamp: "Formatted at 2026-09-27 14:57:05 on dist-test-slave-pz0h"
server_key: "30e9bb62034cb96c89afc201e91d8c5d"
server_key_iv: "88edbfdef408b5c3717b10d2bc52bb0c"
server_key_version: "encryptionkey@0"
I20260927 14:57:05.368463 3924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:05.392657 3924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:05.396034 3924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:05.396230 3924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:05.403384 3924 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40529
I20260927 14:57:05.403409 3991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40529 every 8 connection(s)
I20260927 14:57:05.404387 3924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:57:05.407421 3992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:05.408890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 3924
I20260927 14:57:05.409009 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 14:57:05.409261 392 external_mini_cluster.cc:1428] Setting key 1ac3914829669346a385e82bc337a677
I20260927 14:57:05.413919 3992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5962d34d84d74960ab0fa53f2c4e4a7f: Bootstrap starting.
I20260927 14:57:05.416654 3992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5962d34d84d74960ab0fa53f2c4e4a7f: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:05.417614 3992 log.cc:824] T 00000000000000000000000000000000 P 5962d34d84d74960ab0fa53f2c4e4a7f: Log is configured to *not* fsync() on all Append() calls
Sep 27 14:57:05 dist-test-slave-pz0h krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:57:05.419783 3992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5962d34d84d74960ab0fa53f2c4e4a7f: No bootstrap required, opened a new log
I20260927 14:57:05.423488 3992 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5962d34d84d74960ab0fa53f2c4e4a7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5962d34d84d74960ab0fa53f2c4e4a7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40529 } }
I20260927 14:57:05.423682 3995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:05.410657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54382 (local address 127.0.98.62:40529)
0927 14:57:05.411103 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:05.411119 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:05.411157 (+ 38us) server_negotiation.cc:407] Connection header received
0927 14:57:05.411814 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:05.411833 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:05.412195 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:05.412496 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:05.413420 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:05.414416 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:05.415607 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:05.416110 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:05.416248 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:05.419120 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:05.419125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:05.419139 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:05.419165 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:05.420820 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:05.421235 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:05.421240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:05.421246 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:05.421362 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:05.421634 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:05.421637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:05.421638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:05.422059 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:05.422238 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:05.422533 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 14:57:05.422896 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":146,"threads_started":1}
I20260927 14:57:05.423756 3992 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5962d34d84d74960ab0fa53f2c4e4a7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:05.423820 3992 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5962d34d84d74960ab0fa53f2c4e4a7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5962d34d84d74960ab0fa53f2c4e4a7f, State: Initialized, Role: FOLLOWER
I20260927 14:57:05.424242 3992 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5962d34d84d74960ab0fa53f2c4e4a7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5962d34d84d74960ab0fa53f2c4e4a7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40529 } }
I20260927 14:57:05.424392 3992 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5962d34d84d74960ab0fa53f2c4e4a7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:57:05.424482 3992 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5962d34d84d74960ab0fa53f2c4e4a7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:57:05.424587 3992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5962d34d84d74960ab0fa53f2c4e4a7f [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:05.425541 3992 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5962d34d84d74960ab0fa53f2c4e4a7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5962d34d84d74960ab0fa53f2c4e4a7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40529 } }
I20260927 14:57:05.425915 3992 leader_election.cc:302] T 00000000000000000000000000000000 P 5962d34d84d74960ab0fa53f2c4e4a7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5962d34d84d74960ab0fa53f2c4e4a7f; no voters:
I20260927 14:57:05.426266 3992 leader_election.cc:288] T 00000000000000000000000000000000 P 5962d34d84d74960ab0fa53f2c4e4a7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:57:05.426400 3997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5962d34d84d74960ab0fa53f2c4e4a7f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:57:05.426608 3997 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5962d34d84d74960ab0fa53f2c4e4a7f [term 1 LEADER]: Becoming Leader. State: Replica: 5962d34d84d74960ab0fa53f2c4e4a7f, State: Running, Role: LEADER
I20260927 14:57:05.427002 3997 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5962d34d84d74960ab0fa53f2c4e4a7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5962d34d84d74960ab0fa53f2c4e4a7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40529 } }
I20260927 14:57:05.427541 3992 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5962d34d84d74960ab0fa53f2c4e4a7f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:57:05.428750 3999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5962d34d84d74960ab0fa53f2c4e4a7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5962d34d84d74960ab0fa53f2c4e4a7f. Latest consensus state: current_term: 1 leader_uuid: "5962d34d84d74960ab0fa53f2c4e4a7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5962d34d84d74960ab0fa53f2c4e4a7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40529 } } }
I20260927 14:57:05.428784 3998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5962d34d84d74960ab0fa53f2c4e4a7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5962d34d84d74960ab0fa53f2c4e4a7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5962d34d84d74960ab0fa53f2c4e4a7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40529 } } }
I20260927 14:57:05.428910 3999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5962d34d84d74960ab0fa53f2c4e4a7f [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:05.428910 3998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5962d34d84d74960ab0fa53f2c4e4a7f [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:05.429286 4007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:57:05.432186 4007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:57:05.438038 4007 catalog_manager.cc:1347] Generated new cluster ID: ca883ad7c2b143afbb894cf53ceb892a
I20260927 14:57:05.438119 4007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:57:05.443876 4007 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:57:05.444908 4007 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:57:05.454926 4007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5962d34d84d74960ab0fa53f2c4e4a7f: Generated new TSK 0
I20260927 14:57:05.455768 4007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:57:05.487264 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40529
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:05.609102 4018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:05.609413 4018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:05.609460 4018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:05.614753 4018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:05.614899 4018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:05.614970 4018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:57:05.621001 4018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:05.622124 4018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:05.623214 4018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:05.633181 4026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:05.633297 4018 server_base.cc:1034] running on GCE node
W20260927 14:57:05.633183 4023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:05.633185 4024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:05.633880 4018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:05.634559 4018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:05.635730 4018 hybrid_clock.cc:648] HybridClock initialized: now 1790521025635704 us; error 42 us; skew 500 ppm
Sep 27 14:57:05 dist-test-slave-pz0h krb5kdc[3911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:05.638525 4018 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:57:05.639572 4018 webserver.cc:460] Webserver started at http://127.0.98.1:43737/ using document root <none> and password file <none>
I20260927 14:57:05.640167 4018 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:05.640247 4018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:05.640477 4018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:05.642755 4018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-0/data/instance:
uuid: "1a588dd8a90047fbb774491bf178b9da"
format_stamp: "Formatted at 2026-09-27 14:57:05 on dist-test-slave-pz0h"
server_key: "e0e216d689a1f7141708d09f989e7db8"
server_key_iv: "ce815bf36950f1004d0df39b0e2b2765"
server_key_version: "encryptionkey@0"
I20260927 14:57:05.643420 4018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1a588dd8a90047fbb774491bf178b9da"
format_stamp: "Formatted at 2026-09-27 14:57:05 on dist-test-slave-pz0h"
server_key: "e0e216d689a1f7141708d09f989e7db8"
server_key_iv: "ce815bf36950f1004d0df39b0e2b2765"
server_key_version: "encryptionkey@0"
I20260927 14:57:05.647421 4018 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 14:57:05.650007 4033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:05.651309 4018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:57:05.651440 4018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "1a588dd8a90047fbb774491bf178b9da"
format_stamp: "Formatted at 2026-09-27 14:57:05 on dist-test-slave-pz0h"
server_key: "e0e216d689a1f7141708d09f989e7db8"
server_key_iv: "ce815bf36950f1004d0df39b0e2b2765"
server_key_version: "encryptionkey@0"
I20260927 14:57:05.651549 4018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:05.667938 4018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:05.670959 4018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:05.671142 4018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:05.671747 4018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:05.672633 4018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:05.672683 4018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:05.672737 4018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:05.672763 4018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:05.681416 4018 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36601
I20260927 14:57:05.681432 4146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36601 every 8 connection(s)
I20260927 14:57:05.682350 4018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:57:05.685067 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 4018
I20260927 14:57:05.685196 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:57:05.685612 392 external_mini_cluster.cc:1428] Setting key cac83cfca38bdd3e3d22fab5b2b45792
Sep 27 14:57:05 dist-test-slave-pz0h krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 14:57:05.697816 3995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:05.684301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59225 (local address 127.0.98.62:40529)
0927 14:57:05.684461 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:05.684470 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:05.685505 (+ 1035us) server_negotiation.cc:407] Connection header received
0927 14:57:05.687061 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:05.687064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:05.687113 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:05.687181 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:05.688537 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:05.689300 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:05.691119 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:05.691411 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:05.691514 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:05.694115 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:05.694118 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:05.694120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:05.694149 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:05.695624 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:05.696179 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:05.696182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:05.696183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:05.696267 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:05.696613 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:05.696617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:05.696619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:05.696817 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:05.696918 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:05.697513 (+ 595us) server_negotiation.cc:299] Negotiation successful
0927 14:57:05.697652 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 14:57:05.698644 4149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:05.684711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40529 (local address 127.0.98.1:59225)
0927 14:57:05.685213 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:05.685270 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:05.686837 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:05.687330 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:05.687337 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:05.687711 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:05.688225 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:05.688240 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:05.689487 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:05.689490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:05.690957 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:05.690972 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:05.691601 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:05.691605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:05.691803 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:05.692400 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:05.692419 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:05.693899 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:05.695763 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:05.695770 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:05.695789 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:05.696075 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:05.696355 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:05.696358 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:05.696360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:05.696521 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:05.696922 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:05.696930 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:05.697242 (+ 312us) client_negotiation.cc:769] Sending connection context
0927 14:57:05.697424 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 14:57:05.697722 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":133,"threads_started":1}
I20260927 14:57:05.700474 4147 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40529
I20260927 14:57:05.700814 4147 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:05.701502 4147 heartbeater.cc:511] Master 127.0.98.62:40529 requested a full tablet report, sending...
I20260927 14:57:05.703812 3956 ts_manager.cc:194] Registered new tserver with Master: 1a588dd8a90047fbb774491bf178b9da (127.0.98.1:36601)
I20260927 14:57:05.705461 3956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59225
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:57:05.711819 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40529
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:05.829058 4152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:05.829360 4152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:05.829424 4152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:05.833751 4152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:05.833832 4152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:05.833920 4152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:57:05.839521 4152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:05.840628 4152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:05.841647 4152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:05.851543 4158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:05.851626 4152 server_base.cc:1034] running on GCE node
W20260927 14:57:05.851514 4157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:05.851508 4160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:05.852233 4152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:05.852953 4152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:05.854137 4152 hybrid_clock.cc:648] HybridClock initialized: now 1790521025854102 us; error 47 us; skew 500 ppm
Sep 27 14:57:05 dist-test-slave-pz0h krb5kdc[3911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:05.857417 4152 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:57:05.858599 4152 webserver.cc:460] Webserver started at http://127.0.98.2:42635/ using document root <none> and password file <none>
I20260927 14:57:05.859359 4152 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:05.859453 4152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:05.859704 4152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:05.862511 4152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-1/data/instance:
uuid: "e3d2adf152344c13b9aefa09580fefad"
format_stamp: "Formatted at 2026-09-27 14:57:05 on dist-test-slave-pz0h"
server_key: "0224eba792e43f3971d95820149a7d49"
server_key_iv: "6ae8bae1b57656dd5373c887873ea90a"
server_key_version: "encryptionkey@0"
I20260927 14:57:05.863268 4152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e3d2adf152344c13b9aefa09580fefad"
format_stamp: "Formatted at 2026-09-27 14:57:05 on dist-test-slave-pz0h"
server_key: "0224eba792e43f3971d95820149a7d49"
server_key_iv: "6ae8bae1b57656dd5373c887873ea90a"
server_key_version: "encryptionkey@0"
I20260927 14:57:05.867318 4152 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 14:57:05.869771 4167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:05.871136 4152 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:57:05.871276 4152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "e3d2adf152344c13b9aefa09580fefad"
format_stamp: "Formatted at 2026-09-27 14:57:05 on dist-test-slave-pz0h"
server_key: "0224eba792e43f3971d95820149a7d49"
server_key_iv: "6ae8bae1b57656dd5373c887873ea90a"
server_key_version: "encryptionkey@0"
I20260927 14:57:05.871392 4152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:05.898679 4152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:05.901997 4152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:05.902189 4152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:05.902837 4152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:05.903692 4152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:05.903743 4152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:05.903797 4152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:05.903825 4152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:05.912345 4152 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36193
I20260927 14:57:05.912375 4280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36193 every 8 connection(s)
I20260927 14:57:05.913246 4152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:57:05.914222 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 4152
I20260927 14:57:05.914321 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:57:05.914587 392 external_mini_cluster.cc:1428] Setting key 280ec18db8ce15135bf3720a3eb05763
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 27 14:57:05 dist-test-slave-pz0h krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:57:05.931455 3995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:05.915262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38503 (local address 127.0.98.62:40529)
0927 14:57:05.915393 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:05.915397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:05.916413 (+ 1016us) server_negotiation.cc:407] Connection header received
0927 14:57:05.917795 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:05.917799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:05.917851 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:05.917930 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:05.919416 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:05.920018 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:05.921871 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:05.922109 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:05.922202 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:05.927447 (+ 5245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:05.927452 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:05.927455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:05.927490 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:05.929105 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:05.929796 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:05.929800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:05.929802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:05.929898 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:05.930273 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:05.930275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:05.930276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:05.930477 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:05.930561 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:05.931147 (+ 586us) server_negotiation.cc:299] Negotiation successful
0927 14:57:05.931276 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 14:57:05.932329 4283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:05.915731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40529 (local address 127.0.98.2:38503)
0927 14:57:05.916266 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:05.916313 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:05.917581 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:05.918123 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:05.918131 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:05.918544 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:05.919244 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:05.919263 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:05.920158 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:05.920162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:05.921690 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:05.921707 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:05.922313 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:05.922318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:05.922561 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:05.923353 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:05.923381 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:05.927197 (+ 3816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:05.929254 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:05.929261 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:05.929279 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:05.929680 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:05.929990 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:05.929993 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:05.929995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:05.930185 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:05.930587 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:05.930594 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:05.930896 (+ 302us) client_negotiation.cc:769] Sending connection context
0927 14:57:05.931090 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 14:57:05.931451 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":145,"threads_started":1}
I20260927 14:57:05.933588 4281 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40529
I20260927 14:57:05.933895 4281 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:05.934556 4281 heartbeater.cc:511] Master 127.0.98.62:40529 requested a full tablet report, sending...
I20260927 14:57:05.935767 3956 ts_manager.cc:194] Registered new tserver with Master: e3d2adf152344c13b9aefa09580fefad (127.0.98.2:36193)
I20260927 14:57:05.936359 3956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38503
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:57:05.941745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40529
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:06.054600 4286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:06.054998 4286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:06.055078 4286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:06.059479 4286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:06.059554 4286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:06.059636 4286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:57:06.065237 4286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:06.066388 4286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:06.067474 4286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:06.077239 4292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:06.077255 4291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:06.077255 4294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:06.077450 4286 server_base.cc:1034] running on GCE node
I20260927 14:57:06.077937 4286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:06.078604 4286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:06.079783 4286 hybrid_clock.cc:648] HybridClock initialized: now 1790521026079776 us; error 63 us; skew 500 ppm
Sep 27 14:57:06 dist-test-slave-pz0h krb5kdc[3911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:06.082619 4286 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:57:06.083635 4286 webserver.cc:460] Webserver started at http://127.0.98.3:38455/ using document root <none> and password file <none>
I20260927 14:57:06.084199 4286 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:06.084271 4286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:06.084484 4286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:06.086663 4286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-2/data/instance:
uuid: "2809069ce254443c8e88571aaa62e5f8"
format_stamp: "Formatted at 2026-09-27 14:57:06 on dist-test-slave-pz0h"
server_key: "b331ed540aea6ca7de6870df82039684"
server_key_iv: "9b20963892625d59d2dfc3facb60e8af"
server_key_version: "encryptionkey@0"
I20260927 14:57:06.087387 4286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2809069ce254443c8e88571aaa62e5f8"
format_stamp: "Formatted at 2026-09-27 14:57:06 on dist-test-slave-pz0h"
server_key: "b331ed540aea6ca7de6870df82039684"
server_key_iv: "9b20963892625d59d2dfc3facb60e8af"
server_key_version: "encryptionkey@0"
I20260927 14:57:06.091486 4286 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 14:57:06.094170 4301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:06.095660 4286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:57:06.095798 4286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "2809069ce254443c8e88571aaa62e5f8"
format_stamp: "Formatted at 2026-09-27 14:57:06 on dist-test-slave-pz0h"
server_key: "b331ed540aea6ca7de6870df82039684"
server_key_iv: "9b20963892625d59d2dfc3facb60e8af"
server_key_version: "encryptionkey@0"
I20260927 14:57:06.095917 4286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:06.111768 4286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:06.115000 4286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:06.115202 4286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:06.115811 4286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:06.116657 4286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:06.116705 4286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:06.116760 4286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:06.116786 4286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:06.125301 4286 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40863
I20260927 14:57:06.125327 4414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40863 every 8 connection(s)
I20260927 14:57:06.126210 4286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:57:06.128358 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 4286
I20260927 14:57:06.128491 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790521004237705-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:57:06.128835 392 external_mini_cluster.cc:1428] Setting key 991bc77e20c0468df4425af5a829bcae
Sep 27 14:57:06 dist-test-slave-pz0h krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:57:06.141537 3995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:06.128101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37973 (local address 127.0.98.62:40529)
0927 14:57:06.128240 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:06.128244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:06.129312 (+ 1068us) server_negotiation.cc:407] Connection header received
0927 14:57:06.131173 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:06.131177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:06.131238 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:06.131320 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:06.132778 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:06.133332 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:06.134533 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:06.134757 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:06.134907 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:06.137723 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:06.137726 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:06.137729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:06.137759 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:06.139336 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:06.139923 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:06.139925 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:06.139927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:06.140004 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:06.140331 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:06.140334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:06.140336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:06.140569 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:06.140662 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:06.141270 (+ 608us) server_negotiation.cc:299] Negotiation successful
0927 14:57:06.141400 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 14:57:06.142562 4417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:06.128614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40529 (local address 127.0.98.3:37973)
0927 14:57:06.129141 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:06.129200 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:06.130884 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:06.131535 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:06.131544 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:06.131946 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:06.132566 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:06.132584 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:06.133445 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:06.133448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:06.134395 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:06.134406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:06.134947 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:06.134950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:06.135116 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:06.135808 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:06.135827 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:06.137489 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:06.139483 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:06.139490 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:06.139506 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:06.139826 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:06.140071 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:06.140073 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:06.140074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:06.140226 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:06.140702 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:06.140709 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:06.141009 (+ 300us) client_negotiation.cc:769] Sending connection context
0927 14:57:06.141195 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 14:57:06.141496 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":155,"threads_started":1}
I20260927 14:57:06.143721 4415 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40529
I20260927 14:57:06.143972 4415 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:06.144491 4415 heartbeater.cc:511] Master 127.0.98.62:40529 requested a full tablet report, sending...
I20260927 14:57:06.145651 3956 ts_manager.cc:194] Registered new tserver with Master: 2809069ce254443c8e88571aaa62e5f8 (127.0.98.3:40863)
I20260927 14:57:06.146230 3956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37973
I20260927 14:57:06.155609 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 14:57:06 dist-test-slave-pz0h krb5kdc[3911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 14:57:06 dist-test-slave-pz0h krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 14:57:06.182166 4422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:06.171880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38532 (local address 127.0.98.1:36601)
0927 14:57:06.172200 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:06.172218 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:06.172245 (+ 27us) server_negotiation.cc:407] Connection header received
0927 14:57:06.172317 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:06.172322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:06.172515 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:06.172637 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:06.173332 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:06.174171 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:06.175235 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:06.175670 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:06.175761 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:06.178148 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:06.178160 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:06.178176 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:06.178208 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:06.179828 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:06.180392 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:06.180402 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:06.180403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:06.180580 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:06.180998 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:06.181003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:06.181005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:06.181339 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:06.181559 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:06.181849 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 14:57:06.182005 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":132,"threads_started":1}
Sep 27 14:57:06 dist-test-slave-pz0h krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:57:06.194530 3995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:06.185438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54398 (local address 127.0.98.62:40529)
0927 14:57:06.185656 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:06.185660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:06.185881 (+ 221us) server_negotiation.cc:407] Connection header received
0927 14:57:06.185981 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:06.185984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:06.186038 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:06.186100 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:06.186746 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:06.187343 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:06.188140 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:06.188368 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:06.188456 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:06.191043 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:06.191047 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:06.191050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:06.191086 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:06.192635 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:06.193109 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:06.193113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:06.193114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:06.193207 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:06.193522 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:06.193524 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:06.193526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:06.193772 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:06.193885 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:06.194226 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 14:57:06.194363 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
W20260927 14:57:06.195009 3956 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54398
I20260927 14:57:06.195710 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 4018
I20260927 14:57:06.201773 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 4152
I20260927 14:57:06.207262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 4286
I20260927 14:57:06.212142 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 3924
2026-09-27T14:57:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2071 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:57:06 dist-test-slave-pz0h krb5kdc[4427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:57:06 dist-test-slave-pz0h krb5kdc[4427](info): set up 2 sockets
Sep 27 14:57:06 dist-test-slave-pz0h krb5kdc[4427](info): commencing operation
krb5kdc: starting...
W20260927 14:57:07.162680 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 14:57:07 dist-test-slave-pz0h krb5kdc[4427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:57:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:57:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:57:07.282464 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37377
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:07.397090 4440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:07.397410 4440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:07.397472 4440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:07.401782 4440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:57:07.401876 4440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:07.401903 4440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:07.401924 4440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:57:07.401945 4440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:57:07.407665 4440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:07.408830 4440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:07.409888 4440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:07.418419 4446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:07.418442 4445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:07.418419 4448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:07.418751 4440 server_base.cc:1034] running on GCE node
I20260927 14:57:07.419391 4440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:07.420635 4440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:07.421824 4440 hybrid_clock.cc:648] HybridClock initialized: now 1790521027421801 us; error 47 us; skew 500 ppm
Sep 27 14:57:07 dist-test-slave-pz0h krb5kdc[4427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:07.428874 4440 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:57:07.429714 4440 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:07.429795 4440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:07.430037 4440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:07.432864 4440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "a837f460276c45f682f7f5bf8ade0dc6"
format_stamp: "Formatted at 2026-09-27 14:57:07 on dist-test-slave-pz0h"
server_key: "8c4b81db430eeced065b99dba6d9cbf4"
server_key_iv: "1851abe6052568aea17c687581e51d0b"
server_key_version: "encryptionkey@0"
I20260927 14:57:07.433513 4440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "a837f460276c45f682f7f5bf8ade0dc6"
format_stamp: "Formatted at 2026-09-27 14:57:07 on dist-test-slave-pz0h"
server_key: "8c4b81db430eeced065b99dba6d9cbf4"
server_key_iv: "1851abe6052568aea17c687581e51d0b"
server_key_version: "encryptionkey@0"
I20260927 14:57:07.437705 4440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 14:57:07.440635 4454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:07.442044 4440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:57:07.442189 4440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "a837f460276c45f682f7f5bf8ade0dc6"
format_stamp: "Formatted at 2026-09-27 14:57:07 on dist-test-slave-pz0h"
server_key: "8c4b81db430eeced065b99dba6d9cbf4"
server_key_iv: "1851abe6052568aea17c687581e51d0b"
server_key_version: "encryptionkey@0"
I20260927 14:57:07.442323 4440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:07.456794 4440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:07.460175 4440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:07.460364 4440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:07.467437 4440 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37377
I20260927 14:57:07.467725 4506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37377 every 8 connection(s)
I20260927 14:57:07.468291 4440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:57:07.469229 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 4440
I20260927 14:57:07.469362 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 14:57:07.469662 392 external_mini_cluster.cc:1428] Setting key a661abf16924c6c72c71b3f18cf3e1de
I20260927 14:57:07.472240 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:07.479270 4507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a837f460276c45f682f7f5bf8ade0dc6: Bootstrap starting.
Sep 27 14:57:07 dist-test-slave-pz0h krb5kdc[4427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:57:07.481775 4507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a837f460276c45f682f7f5bf8ade0dc6: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:07.482532 4507 log.cc:824] T 00000000000000000000000000000000 P a837f460276c45f682f7f5bf8ade0dc6: Log is configured to *not* fsync() on all Append() calls
I20260927 14:57:07.484594 4507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a837f460276c45f682f7f5bf8ade0dc6: No bootstrap required, opened a new log
I20260927 14:57:07.485180 4510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:07.471404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46058 (local address 127.0.98.62:37377)
0927 14:57:07.471916 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:07.471931 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:07.472144 (+ 213us) server_negotiation.cc:407] Connection header received
0927 14:57:07.472972 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:07.472990 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:07.473348 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:07.473661 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:07.474537 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:07.475627 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:07.476772 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:07.477282 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:07.477421 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:07.480185 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:07.480191 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:07.480205 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:07.480236 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:07.481907 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:07.482337 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:07.482343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:07.482348 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:07.482811 (+ 463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:07.483138 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:07.483142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:07.483144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:07.483581 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:07.483769 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:07.484061 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 14:57:07.484350 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":165,"threads_started":1}
I20260927 14:57:07.488591 4507 raft_consensus.cc:348] T 00000000000000000000000000000000 P a837f460276c45f682f7f5bf8ade0dc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a837f460276c45f682f7f5bf8ade0dc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37377 } }
I20260927 14:57:07.488863 4507 raft_consensus.cc:374] T 00000000000000000000000000000000 P a837f460276c45f682f7f5bf8ade0dc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:07.488924 4507 raft_consensus.cc:729] T 00000000000000000000000000000000 P a837f460276c45f682f7f5bf8ade0dc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a837f460276c45f682f7f5bf8ade0dc6, State: Initialized, Role: FOLLOWER
I20260927 14:57:07.489388 4507 consensus_queue.cc:260] T 00000000000000000000000000000000 P a837f460276c45f682f7f5bf8ade0dc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a837f460276c45f682f7f5bf8ade0dc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37377 } }
I20260927 14:57:07.489528 4507 raft_consensus.cc:388] T 00000000000000000000000000000000 P a837f460276c45f682f7f5bf8ade0dc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:57:07.489598 4507 raft_consensus.cc:482] T 00000000000000000000000000000000 P a837f460276c45f682f7f5bf8ade0dc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:57:07.489701 4507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a837f460276c45f682f7f5bf8ade0dc6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:07.490911 4507 raft_consensus.cc:504] T 00000000000000000000000000000000 P a837f460276c45f682f7f5bf8ade0dc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a837f460276c45f682f7f5bf8ade0dc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37377 } }
I20260927 14:57:07.491278 4507 leader_election.cc:302] T 00000000000000000000000000000000 P a837f460276c45f682f7f5bf8ade0dc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a837f460276c45f682f7f5bf8ade0dc6; no voters:
I20260927 14:57:07.491616 4507 leader_election.cc:288] T 00000000000000000000000000000000 P a837f460276c45f682f7f5bf8ade0dc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:57:07.491779 4512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a837f460276c45f682f7f5bf8ade0dc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:57:07.492007 4512 raft_consensus.cc:686] T 00000000000000000000000000000000 P a837f460276c45f682f7f5bf8ade0dc6 [term 1 LEADER]: Becoming Leader. State: Replica: a837f460276c45f682f7f5bf8ade0dc6, State: Running, Role: LEADER
I20260927 14:57:07.492340 4512 consensus_queue.cc:237] T 00000000000000000000000000000000 P a837f460276c45f682f7f5bf8ade0dc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a837f460276c45f682f7f5bf8ade0dc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37377 } }
I20260927 14:57:07.492889 4507 sys_catalog.cc:564] T 00000000000000000000000000000000 P a837f460276c45f682f7f5bf8ade0dc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:57:07.494009 4514 sys_catalog.cc:455] T 00000000000000000000000000000000 P a837f460276c45f682f7f5bf8ade0dc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a837f460276c45f682f7f5bf8ade0dc6. Latest consensus state: current_term: 1 leader_uuid: "a837f460276c45f682f7f5bf8ade0dc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a837f460276c45f682f7f5bf8ade0dc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37377 } } }
I20260927 14:57:07.494148 4514 sys_catalog.cc:458] T 00000000000000000000000000000000 P a837f460276c45f682f7f5bf8ade0dc6 [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:07.494446 4513 sys_catalog.cc:455] T 00000000000000000000000000000000 P a837f460276c45f682f7f5bf8ade0dc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a837f460276c45f682f7f5bf8ade0dc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a837f460276c45f682f7f5bf8ade0dc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37377 } } }
I20260927 14:57:07.494611 4513 sys_catalog.cc:458] T 00000000000000000000000000000000 P a837f460276c45f682f7f5bf8ade0dc6 [sys.catalog]: This master's current role is: LEADER
W20260927 14:57:07.497452 4527 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a837f460276c45f682f7f5bf8ade0dc6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:57:07.497534 4527 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:57:07.497620 4521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:57:07.498333 4521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:57:07.504946 4521 catalog_manager.cc:1347] Generated new cluster ID: 2701c08735434915bc5271ea2892848e
I20260927 14:57:07.505050 4521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:57:07.515730 4521 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:57:07.517143 4521 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:57:07.523993 4521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a837f460276c45f682f7f5bf8ade0dc6: Generated new TSK 0
I20260927 14:57:07.524824 4521 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:57:07.560024 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37377
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:07.679379 4533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:07.679689 4533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:07.679756 4533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:07.684073 4533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:07.684163 4533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:07.684252 4533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:57:07.690028 4533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:07.691207 4533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:07.692273 4533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:07.702400 4539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:07.702453 4533 server_base.cc:1034] running on GCE node
W20260927 14:57:07.702387 4538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:07.702389 4541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:07.703111 4533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:07.703760 4533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:07.704949 4533 hybrid_clock.cc:648] HybridClock initialized: now 1790521027704920 us; error 42 us; skew 500 ppm
Sep 27 14:57:07 dist-test-slave-pz0h krb5kdc[4427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:07.708189 4533 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:57:07.709024 4533 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:07.709100 4533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:07.709307 4533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:07.711515 4533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-0/data/instance:
uuid: "a1e61523dc02493e913aaaf41a46c44b"
format_stamp: "Formatted at 2026-09-27 14:57:07 on dist-test-slave-pz0h"
server_key: "869a1e32454373c9edc7a25cc5449cdf"
server_key_iv: "30a416961f8c213b4c5df105ec54099a"
server_key_version: "encryptionkey@0"
I20260927 14:57:07.712141 4533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a1e61523dc02493e913aaaf41a46c44b"
format_stamp: "Formatted at 2026-09-27 14:57:07 on dist-test-slave-pz0h"
server_key: "869a1e32454373c9edc7a25cc5449cdf"
server_key_iv: "30a416961f8c213b4c5df105ec54099a"
server_key_version: "encryptionkey@0"
I20260927 14:57:07.716084 4533 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 14:57:07.718755 4547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:07.720216 4533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:57:07.720367 4533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "a1e61523dc02493e913aaaf41a46c44b"
format_stamp: "Formatted at 2026-09-27 14:57:07 on dist-test-slave-pz0h"
server_key: "869a1e32454373c9edc7a25cc5449cdf"
server_key_iv: "30a416961f8c213b4c5df105ec54099a"
server_key_version: "encryptionkey@0"
I20260927 14:57:07.720494 4533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:07.747123 4533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:07.750403 4533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:07.750598 4533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:07.751228 4533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:07.752203 4533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:07.752267 4533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:07.752327 4533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:07.752355 4533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:07.762024 4533 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41595
I20260927 14:57:07.762045 4660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41595 every 8 connection(s)
I20260927 14:57:07.762666 4533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:57:07.767377 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 4533
I20260927 14:57:07.767509 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:57:07.767807 392 external_mini_cluster.cc:1428] Setting key acb034186f6959e3c7ed8876ef6eb6f5
Sep 27 14:57:07 dist-test-slave-pz0h krb5kdc[4427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 14:57:07.778782 4510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:07.764568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39525 (local address 127.0.98.62:37377)
0927 14:57:07.764721 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:07.764726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:07.765619 (+ 893us) server_negotiation.cc:407] Connection header received
0927 14:57:07.767048 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:07.767051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:07.767105 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:07.767192 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:07.768760 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:07.769447 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:07.770774 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:07.771107 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:07.771208 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:07.773961 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:07.773964 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:07.773966 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:07.773993 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:07.775749 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:07.776559 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:07.776563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:07.776565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:07.776659 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:07.777066 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:07.777069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:07.777135 (+ 66us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:07.777576 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:07.777793 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:07.778474 (+ 681us) server_negotiation.cc:299] Negotiation successful
0927 14:57:07.778640 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 14:57:07.779865 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:07.764941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37377 (local address 127.0.98.1:39525)
0927 14:57:07.765446 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:07.765495 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:07.766747 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:07.767447 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:07.767458 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:07.767982 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:07.768598 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:07.768618 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:07.769593 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:07.769597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:07.770620 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:07.770642 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:07.771279 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:07.771284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:07.771492 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:07.772211 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:07.772229 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:07.773755 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:07.776005 (+ 2250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:07.776013 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:07.776035 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:07.776437 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:07.776761 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:07.776765 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:07.776767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:07.776970 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:07.777775 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:07.777784 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:07.778144 (+ 360us) client_negotiation.cc:769] Sending connection context
0927 14:57:07.778416 (+ 272us) client_negotiation.cc:240] Negotiation successful
0927 14:57:07.778777 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":123,"threads_started":1}
I20260927 14:57:07.781363 4661 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37377
I20260927 14:57:07.781684 4661 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:07.782224 4661 heartbeater.cc:511] Master 127.0.98.62:37377 requested a full tablet report, sending...
I20260927 14:57:07.784054 4471 ts_manager.cc:194] Registered new tserver with Master: a1e61523dc02493e913aaaf41a46c44b (127.0.98.1:41595)
I20260927 14:57:07.785215 4471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:57:07.794291 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37377
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:07.908179 4666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:07.908484 4666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:07.908550 4666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:07.912804 4666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:07.912889 4666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:07.912974 4666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:57:07.918752 4666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:07.919924 4666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:07.920959 4666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:07.930948 4674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:07.930962 4671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:07.931011 4666 server_base.cc:1034] running on GCE node
W20260927 14:57:07.930965 4672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:07.931623 4666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:07.932303 4666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:07.933452 4666 hybrid_clock.cc:648] HybridClock initialized: now 1790521027933425 us; error 44 us; skew 500 ppm
Sep 27 14:57:07 dist-test-slave-pz0h krb5kdc[4427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:07.936240 4666 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:57:07.936983 4666 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:07.937059 4666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:07.937305 4666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:07.939551 4666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-1/data/instance:
uuid: "80c231f81022425692df97e6f3e1fbee"
format_stamp: "Formatted at 2026-09-27 14:57:07 on dist-test-slave-pz0h"
server_key: "fa1a2798b2e6b36821055eb0c552ff68"
server_key_iv: "b6060ca047f5b25a4777a1806b5fff58"
server_key_version: "encryptionkey@0"
I20260927 14:57:07.940141 4666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "80c231f81022425692df97e6f3e1fbee"
format_stamp: "Formatted at 2026-09-27 14:57:07 on dist-test-slave-pz0h"
server_key: "fa1a2798b2e6b36821055eb0c552ff68"
server_key_iv: "b6060ca047f5b25a4777a1806b5fff58"
server_key_version: "encryptionkey@0"
I20260927 14:57:07.944072 4666 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 14:57:07.946825 4680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:07.948215 4666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:57:07.948338 4666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "80c231f81022425692df97e6f3e1fbee"
format_stamp: "Formatted at 2026-09-27 14:57:07 on dist-test-slave-pz0h"
server_key: "fa1a2798b2e6b36821055eb0c552ff68"
server_key_iv: "b6060ca047f5b25a4777a1806b5fff58"
server_key_version: "encryptionkey@0"
I20260927 14:57:07.948452 4666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:07.959520 4666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:07.962975 4666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:07.963169 4666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:07.963750 4666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:07.964594 4666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:07.964641 4666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:07.964696 4666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:07.964721 4666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:07.973424 4666 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33799
I20260927 14:57:07.973455 4793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33799 every 8 connection(s)
I20260927 14:57:07.974028 4666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:57:07.980870 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 4666
I20260927 14:57:07.981011 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:57:07.981297 392 external_mini_cluster.cc:1428] Setting key d0300db298cc99420b2f749aef78d542
Sep 27 14:57:07 dist-test-slave-pz0h krb5kdc[4427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:57:07.988104 4510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:07.975883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32895 (local address 127.0.98.62:37377)
0927 14:57:07.976052 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:07.976056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:07.976912 (+ 856us) server_negotiation.cc:407] Connection header received
0927 14:57:07.978299 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:07.978303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:07.978361 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:07.978435 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:07.979696 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:07.980229 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:07.981433 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:07.981658 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:07.981750 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:07.984502 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:07.984507 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:07.984510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:07.984542 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:07.985971 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:07.986493 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:07.986497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:07.986498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:07.986573 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:07.986949 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:07.986952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:07.986954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:07.987146 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:07.987235 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:07.987790 (+ 555us) server_negotiation.cc:299] Negotiation successful
0927 14:57:07.987933 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 14:57:07.988965 4796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:07.976242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37377 (local address 127.0.98.2:32895)
0927 14:57:07.976727 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:07.976773 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:07.978057 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:07.978624 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:07.978631 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:07.979041 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:07.979545 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:07.979559 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:07.980362 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:07.980365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:07.981307 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:07.981317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:07.981814 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:07.981819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:07.981992 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:07.982587 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:07.982604 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:07.984270 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:07.986097 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:07.986102 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:07.986117 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:07.986386 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:07.986663 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:07.986665 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:07.986667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:07.986854 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:07.987250 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:07.987257 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:07.987536 (+ 279us) client_negotiation.cc:769] Sending connection context
0927 14:57:07.987708 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 14:57:07.988075 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":120,"threads_started":1}
I20260927 14:57:07.990181 4794 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37377
I20260927 14:57:07.990475 4794 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 14:57:07.991086 4794 heartbeater.cc:511] Master 127.0.98.62:37377 requested a full tablet report, sending...
I20260927 14:57:07.992286 4471 ts_manager.cc:194] Registered new tserver with Master: 80c231f81022425692df97e6f3e1fbee (127.0.98.2:33799)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 14:57:07.992988 4471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:32895
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:57:08.007463 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37377
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:08.121850 4799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:08.122133 4799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:08.122179 4799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:08.126436 4799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:08.126513 4799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:08.126577 4799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:57:08.132112 4799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:08.133174 4799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:08.134166 4799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:08.143642 4804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:08.143678 4799 server_base.cc:1034] running on GCE node
W20260927 14:57:08.143612 4805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:08.143606 4807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:08.144347 4799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:08.144999 4799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:08.146198 4799 hybrid_clock.cc:648] HybridClock initialized: now 1790521028146176 us; error 38 us; skew 500 ppm
Sep 27 14:57:08 dist-test-slave-pz0h krb5kdc[4427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:08.149304 4799 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:57:08.150087 4799 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:08.150161 4799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:08.150357 4799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:08.152616 4799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-2/data/instance:
uuid: "62c6ee18735e40acb8a0a198533d83e0"
format_stamp: "Formatted at 2026-09-27 14:57:08 on dist-test-slave-pz0h"
server_key: "7061e7903a4e10ad4602cdf9c56b02f7"
server_key_iv: "6897e6e212e1b6e5ba0af19816ad9acc"
server_key_version: "encryptionkey@0"
I20260927 14:57:08.153270 4799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "62c6ee18735e40acb8a0a198533d83e0"
format_stamp: "Formatted at 2026-09-27 14:57:08 on dist-test-slave-pz0h"
server_key: "7061e7903a4e10ad4602cdf9c56b02f7"
server_key_iv: "6897e6e212e1b6e5ba0af19816ad9acc"
server_key_version: "encryptionkey@0"
I20260927 14:57:08.157191 4799 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 14:57:08.159729 4813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:08.161062 4799 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:57:08.161194 4799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "62c6ee18735e40acb8a0a198533d83e0"
format_stamp: "Formatted at 2026-09-27 14:57:08 on dist-test-slave-pz0h"
server_key: "7061e7903a4e10ad4602cdf9c56b02f7"
server_key_iv: "6897e6e212e1b6e5ba0af19816ad9acc"
server_key_version: "encryptionkey@0"
I20260927 14:57:08.161311 4799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:08.180961 4799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:08.184283 4799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:08.184479 4799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:08.185088 4799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:08.185967 4799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:08.186017 4799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:08.186071 4799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:08.186095 4799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:08.195391 4799 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34077
I20260927 14:57:08.195448 4926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34077 every 8 connection(s)
I20260927 14:57:08.195996 4799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:57:08.204391 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 4799
I20260927 14:57:08.204505 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:57:08.204780 392 external_mini_cluster.cc:1428] Setting key 5a4bcdba10643a876c28e7d3ef4128dd
Sep 27 14:57:08 dist-test-slave-pz0h krb5kdc[4427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:57:08.209319 4510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:08.197716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50669 (local address 127.0.98.62:37377)
0927 14:57:08.197862 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:08.197866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:08.198660 (+ 794us) server_negotiation.cc:407] Connection header received
0927 14:57:08.200015 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:08.200018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:08.200074 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:08.200146 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:08.201283 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:08.201801 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:08.202984 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:08.203202 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:08.203290 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:08.205920 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:08.205923 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:08.205925 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:08.205954 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:08.207351 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:08.207886 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:08.207889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:08.207890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:08.207969 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:08.208285 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:08.208288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:08.208290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:08.208489 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:08.208573 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:08.209077 (+ 504us) server_negotiation.cc:299] Negotiation successful
0927 14:57:08.209196 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:57:08.210150 4929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:08.198072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37377 (local address 127.0.98.3:50669)
0927 14:57:08.198515 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:08.198558 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:08.199803 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:08.200310 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:08.200318 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:08.200681 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:08.201150 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:08.201162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:08.201927 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:08.201930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:08.202874 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:08.202885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:08.203349 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:08.203353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:08.203521 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:08.204123 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:08.204140 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:08.205740 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:08.207475 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:08.207481 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:08.207495 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:08.207769 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:08.208050 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:08.208052 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:08.208053 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:08.208187 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:08.208571 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:08.208576 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:08.208837 (+ 261us) client_negotiation.cc:769] Sending connection context
0927 14:57:08.208997 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 14:57:08.209265 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":113,"threads_started":1}
I20260927 14:57:08.211395 4927 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37377
I20260927 14:57:08.211613 4927 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:08.212037 4927 heartbeater.cc:511] Master 127.0.98.62:37377 requested a full tablet report, sending...
I20260927 14:57:08.213004 4471 ts_manager.cc:194] Registered new tserver with Master: 62c6ee18735e40acb8a0a198533d83e0 (127.0.98.3:34077)
I20260927 14:57:08.213541 4471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50669
I20260927 14:57:08.218724 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:57:08.228152 4510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:08.220603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46072 (local address 127.0.98.62:37377)
0927 14:57:08.220734 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:08.220738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:08.220886 (+ 148us) server_negotiation.cc:407] Connection header received
0927 14:57:08.221006 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:08.221008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:08.221053 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:08.221119 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:08.221809 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:08.222348 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:08.223292 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:08.223526 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:08.223615 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:08.225094 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:08.225097 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:08.225100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:08.225123 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:08.226530 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:08.227005 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:08.227009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:08.227011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:08.227104 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:08.227353 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:08.227355 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:08.227356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:08.227578 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:08.227686 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:08.227904 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 14:57:08.228023 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:57:08.231199 4471 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:57:08.234268 4471 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:57:08.249781 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:08.243430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41595 (local address 127.0.0.1:45822)
0927 14:57:08.244306 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:08.244360 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:08.244493 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:08.245120 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:08.245126 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:08.245148 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:08.245291 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:08.245298 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:08.246717 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:08.246721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:08.248289 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:08.248300 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:08.249245 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:08.249249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:08.249350 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:08.249388 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 14:57:08.249519 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 14:57:08.249644 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"mutex_wait_us":110,"thread_start_us":108,"threads_started":1}
I20260927 14:57:08.249954 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:08.243576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45822 (local address 127.0.98.1:41595)
0927 14:57:08.244594 (+ 1018us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:08.244600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:08.244624 (+ 24us) server_negotiation.cc:407] Connection header received
0927 14:57:08.244702 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:08.244707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:08.244927 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:08.245129 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:08.245403 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:08.246595 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:08.248423 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:08.249084 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:08.249204 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:08.249369 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:08.249562 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 14:57:08.249680 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":924,"thread_start_us":58,"threads_started":1}
I20260927 14:57:08.251617 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:08.244036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55718 (local address 127.0.98.2:33799)
0927 14:57:08.244423 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:08.244436 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:08.244704 (+ 268us) server_negotiation.cc:407] Connection header received
0927 14:57:08.244783 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:08.244788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:08.244991 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:08.245133 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:08.245533 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:08.246510 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:08.249045 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:08.249854 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:08.249970 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:08.250137 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:08.251331 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0927 14:57:08.251448 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":103,"threads_started":1}
I20260927 14:57:08.251799 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:08.244378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40256 (local address 127.0.98.3:34077)
0927 14:57:08.245159 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:08.245170 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:08.245191 (+ 21us) server_negotiation.cc:407] Connection header received
0927 14:57:08.245257 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:08.245262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:08.245416 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:08.245606 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:08.245755 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:08.246866 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:08.250218 (+ 3352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:08.250924 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:08.251347 (+ 423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:08.251513 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:08.251565 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 14:57:08.251658 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":149,"threads_started":1}
I20260927 14:57:08.251969 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:08.243531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33799 (local address 127.0.0.1:55718)
0927 14:57:08.244504 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:08.244519 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:08.244612 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:08.245303 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:08.245306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:08.245322 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:08.245405 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:08.245411 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:08.246657 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:08.246661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:08.248801 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:08.248829 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:08.250891 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:08.250895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:08.250990 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:08.251029 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 14:57:08.251098 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 14:57:08.251194 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"mutex_wait_us":70,"thread_start_us":121,"threads_started":1}
I20260927 14:57:08.251968 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:08.244221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34077 (local address 127.0.0.1:40256)
0927 14:57:08.244714 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:08.244725 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:08.244789 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:08.245579 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:08.245583 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:08.245599 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:08.245677 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:08.245683 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:08.246958 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:08.246961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:08.248800 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:08.248829 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:08.251063 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:08.251066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:08.251148 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:08.251167 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 14:57:08.251208 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 14:57:08.251269 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":117,"thread_start_us":90,"threads_started":1}
I20260927 14:57:08.253317 4595 tablet_service.cc:1467] Processing CreateTablet for tablet e66a96b2d7734ac7b453a733f638317f (DEFAULT_TABLE table=test-table [id=feb213108f014ff4976913130de2e6a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:57:08.254343 4595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e66a96b2d7734ac7b453a733f638317f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:08.254968 4861 tablet_service.cc:1467] Processing CreateTablet for tablet e66a96b2d7734ac7b453a733f638317f (DEFAULT_TABLE table=test-table [id=feb213108f014ff4976913130de2e6a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:57:08.254944 4728 tablet_service.cc:1467] Processing CreateTablet for tablet e66a96b2d7734ac7b453a733f638317f (DEFAULT_TABLE table=test-table [id=feb213108f014ff4976913130de2e6a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:57:08.256103 4861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e66a96b2d7734ac7b453a733f638317f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:08.256098 4728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e66a96b2d7734ac7b453a733f638317f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:08.261448 4944 tablet_bootstrap.cc:492] T e66a96b2d7734ac7b453a733f638317f P a1e61523dc02493e913aaaf41a46c44b: Bootstrap starting.
I20260927 14:57:08.263906 4944 tablet_bootstrap.cc:654] T e66a96b2d7734ac7b453a733f638317f P a1e61523dc02493e913aaaf41a46c44b: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:08.263979 4945 tablet_bootstrap.cc:492] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0: Bootstrap starting.
I20260927 14:57:08.265231 4946 tablet_bootstrap.cc:492] T e66a96b2d7734ac7b453a733f638317f P 80c231f81022425692df97e6f3e1fbee: Bootstrap starting.
I20260927 14:57:08.265384 4944 log.cc:824] T e66a96b2d7734ac7b453a733f638317f P a1e61523dc02493e913aaaf41a46c44b: Log is configured to *not* fsync() on all Append() calls
I20260927 14:57:08.267489 4944 tablet_bootstrap.cc:492] T e66a96b2d7734ac7b453a733f638317f P a1e61523dc02493e913aaaf41a46c44b: No bootstrap required, opened a new log
I20260927 14:57:08.267699 4944 ts_tablet_manager.cc:1397] T e66a96b2d7734ac7b453a733f638317f P a1e61523dc02493e913aaaf41a46c44b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 14:57:08.267776 4946 tablet_bootstrap.cc:654] T e66a96b2d7734ac7b453a733f638317f P 80c231f81022425692df97e6f3e1fbee: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:08.267776 4945 tablet_bootstrap.cc:654] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:08.269134 4946 log.cc:824] T e66a96b2d7734ac7b453a733f638317f P 80c231f81022425692df97e6f3e1fbee: Log is configured to *not* fsync() on all Append() calls
I20260927 14:57:08.269134 4945 log.cc:824] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0: Log is configured to *not* fsync() on all Append() calls
I20260927 14:57:08.270973 4944 raft_consensus.cc:348] T e66a96b2d7734ac7b453a733f638317f P a1e61523dc02493e913aaaf41a46c44b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e61523dc02493e913aaaf41a46c44b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41595 } } peers { permanent_uuid: "80c231f81022425692df97e6f3e1fbee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33799 } } peers { permanent_uuid: "62c6ee18735e40acb8a0a198533d83e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34077 } }
I20260927 14:57:08.271220 4944 raft_consensus.cc:374] T e66a96b2d7734ac7b453a733f638317f P a1e61523dc02493e913aaaf41a46c44b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:08.271281 4944 raft_consensus.cc:729] T e66a96b2d7734ac7b453a733f638317f P a1e61523dc02493e913aaaf41a46c44b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1e61523dc02493e913aaaf41a46c44b, State: Initialized, Role: FOLLOWER
I20260927 14:57:08.271389 4945 tablet_bootstrap.cc:492] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0: No bootstrap required, opened a new log
I20260927 14:57:08.271389 4946 tablet_bootstrap.cc:492] T e66a96b2d7734ac7b453a733f638317f P 80c231f81022425692df97e6f3e1fbee: No bootstrap required, opened a new log
I20260927 14:57:08.271602 4945 ts_tablet_manager.cc:1397] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 14:57:08.271602 4946 ts_tablet_manager.cc:1397] T e66a96b2d7734ac7b453a733f638317f P 80c231f81022425692df97e6f3e1fbee: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 14:57:08.271778 4944 consensus_queue.cc:260] T e66a96b2d7734ac7b453a733f638317f P a1e61523dc02493e913aaaf41a46c44b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e61523dc02493e913aaaf41a46c44b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41595 } } peers { permanent_uuid: "80c231f81022425692df97e6f3e1fbee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33799 } } peers { permanent_uuid: "62c6ee18735e40acb8a0a198533d83e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34077 } }
I20260927 14:57:08.272444 4661 heartbeater.cc:503] Master 127.0.98.62:37377 was elected leader, sending a full tablet report...
I20260927 14:57:08.272832 4944 ts_tablet_manager.cc:1428] T e66a96b2d7734ac7b453a733f638317f P a1e61523dc02493e913aaaf41a46c44b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 14:57:08.275556 4945 raft_consensus.cc:348] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e61523dc02493e913aaaf41a46c44b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41595 } } peers { permanent_uuid: "80c231f81022425692df97e6f3e1fbee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33799 } } peers { permanent_uuid: "62c6ee18735e40acb8a0a198533d83e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34077 } }
I20260927 14:57:08.275877 4945 raft_consensus.cc:374] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:08.275939 4945 raft_consensus.cc:729] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62c6ee18735e40acb8a0a198533d83e0, State: Initialized, Role: FOLLOWER
I20260927 14:57:08.276423 4945 consensus_queue.cc:260] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e61523dc02493e913aaaf41a46c44b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41595 } } peers { permanent_uuid: "80c231f81022425692df97e6f3e1fbee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33799 } } peers { permanent_uuid: "62c6ee18735e40acb8a0a198533d83e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34077 } }
I20260927 14:57:08.276544 4946 raft_consensus.cc:348] T e66a96b2d7734ac7b453a733f638317f P 80c231f81022425692df97e6f3e1fbee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e61523dc02493e913aaaf41a46c44b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41595 } } peers { permanent_uuid: "80c231f81022425692df97e6f3e1fbee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33799 } } peers { permanent_uuid: "62c6ee18735e40acb8a0a198533d83e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34077 } }
I20260927 14:57:08.276857 4946 raft_consensus.cc:374] T e66a96b2d7734ac7b453a733f638317f P 80c231f81022425692df97e6f3e1fbee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:08.276944 4946 raft_consensus.cc:729] T e66a96b2d7734ac7b453a733f638317f P 80c231f81022425692df97e6f3e1fbee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80c231f81022425692df97e6f3e1fbee, State: Initialized, Role: FOLLOWER
I20260927 14:57:08.277233 4927 heartbeater.cc:503] Master 127.0.98.62:37377 was elected leader, sending a full tablet report...
I20260927 14:57:08.277403 4946 consensus_queue.cc:260] T e66a96b2d7734ac7b453a733f638317f P 80c231f81022425692df97e6f3e1fbee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e61523dc02493e913aaaf41a46c44b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41595 } } peers { permanent_uuid: "80c231f81022425692df97e6f3e1fbee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33799 } } peers { permanent_uuid: "62c6ee18735e40acb8a0a198533d83e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34077 } }
I20260927 14:57:08.278060 4794 heartbeater.cc:503] Master 127.0.98.62:37377 was elected leader, sending a full tablet report...
I20260927 14:57:08.278412 4946 ts_tablet_manager.cc:1428] T e66a96b2d7734ac7b453a733f638317f P 80c231f81022425692df97e6f3e1fbee: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 14:57:08.278497 4945 ts_tablet_manager.cc:1428] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 14:57:08.298715 4951 raft_consensus.cc:482] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:57:08.298980 4951 raft_consensus.cc:504] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e61523dc02493e913aaaf41a46c44b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41595 } } peers { permanent_uuid: "80c231f81022425692df97e6f3e1fbee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33799 } } peers { permanent_uuid: "62c6ee18735e40acb8a0a198533d83e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34077 } }
I20260927 14:57:08.300127 4951 leader_election.cc:288] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1e61523dc02493e913aaaf41a46c44b (127.0.98.1:41595), 80c231f81022425692df97e6f3e1fbee (127.0.98.2:33799)
I20260927 14:57:08.305202 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:08.300411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44291 (local address 127.0.98.1:41595)
0927 14:57:08.300534 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:08.300538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:08.300659 (+ 121us) server_negotiation.cc:407] Connection header received
0927 14:57:08.300748 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:08.300751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:08.300797 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:08.300889 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:08.301135 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:08.301937 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:08.303855 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:08.304510 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:08.304635 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:08.304706 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:08.304973 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 14:57:08.305058 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 14:57:08.305202 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:08.300546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50465 (local address 127.0.98.2:33799)
0927 14:57:08.300668 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:08.300673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:08.300816 (+ 143us) server_negotiation.cc:407] Connection header received
0927 14:57:08.300887 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:08.300890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:08.300936 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:08.301003 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:08.301233 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:08.301962 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:08.303855 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:08.304510 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:08.304635 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:08.304706 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:08.304973 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 14:57:08.305070 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 14:57:08.305244 4929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:08.300354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41595 (local address 127.0.98.3:44291)
0927 14:57:08.300543 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:08.300557 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:08.300659 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:08.300906 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:08.300908 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:08.300921 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:08.301030 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:08.301036 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:08.302112 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:08.302116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:08.303664 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:08.303680 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:08.304679 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:08.304683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:08.304836 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:08.304852 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 14:57:08.304952 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 14:57:08.305080 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":162}
I20260927 14:57:08.305258 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:08.300377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33799 (local address 127.0.98.3:50465)
0927 14:57:08.300727 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:08.300739 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:08.300816 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:08.301029 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:08.301032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:08.301045 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:08.301139 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:08.301145 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:08.302112 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:08.302116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:08.303664 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:08.303680 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:08.304679 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:08.304683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:08.304822 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:08.304852 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 14:57:08.304943 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 14:57:08.305080 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":162,"thread_start_us":123,"threads_started":1}
I20260927 14:57:08.306068 4615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e66a96b2d7734ac7b453a733f638317f" candidate_uuid: "62c6ee18735e40acb8a0a198533d83e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1e61523dc02493e913aaaf41a46c44b" is_pre_election: true
I20260927 14:57:08.306255 4748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e66a96b2d7734ac7b453a733f638317f" candidate_uuid: "62c6ee18735e40acb8a0a198533d83e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80c231f81022425692df97e6f3e1fbee" is_pre_election: true
I20260927 14:57:08.306417 4615 raft_consensus.cc:2454] T e66a96b2d7734ac7b453a733f638317f P a1e61523dc02493e913aaaf41a46c44b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62c6ee18735e40acb8a0a198533d83e0 in term 0.
I20260927 14:57:08.306579 4748 raft_consensus.cc:2454] T e66a96b2d7734ac7b453a733f638317f P 80c231f81022425692df97e6f3e1fbee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62c6ee18735e40acb8a0a198533d83e0 in term 0.
I20260927 14:57:08.307159 4817 leader_election.cc:302] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c6ee18735e40acb8a0a198533d83e0, a1e61523dc02493e913aaaf41a46c44b; no voters:
I20260927 14:57:08.307474 4951 raft_consensus.cc:2781] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:57:08.307555 4951 raft_consensus.cc:482] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:57:08.307586 4951 raft_consensus.cc:3037] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:08.308660 4951 raft_consensus.cc:504] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e61523dc02493e913aaaf41a46c44b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41595 } } peers { permanent_uuid: "80c231f81022425692df97e6f3e1fbee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33799 } } peers { permanent_uuid: "62c6ee18735e40acb8a0a198533d83e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34077 } }
I20260927 14:57:08.309032 4951 leader_election.cc:288] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0 [CANDIDATE]: Term 1 election: Requested vote from peers a1e61523dc02493e913aaaf41a46c44b (127.0.98.1:41595), 80c231f81022425692df97e6f3e1fbee (127.0.98.2:33799)
I20260927 14:57:08.309453 4615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e66a96b2d7734ac7b453a733f638317f" candidate_uuid: "62c6ee18735e40acb8a0a198533d83e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1e61523dc02493e913aaaf41a46c44b"
I20260927 14:57:08.309583 4615 raft_consensus.cc:3037] T e66a96b2d7734ac7b453a733f638317f P a1e61523dc02493e913aaaf41a46c44b [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:08.309576 4748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e66a96b2d7734ac7b453a733f638317f" candidate_uuid: "62c6ee18735e40acb8a0a198533d83e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80c231f81022425692df97e6f3e1fbee"
I20260927 14:57:08.309901 4748 raft_consensus.cc:3037] T e66a96b2d7734ac7b453a733f638317f P 80c231f81022425692df97e6f3e1fbee [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:08.310688 4615 raft_consensus.cc:2454] T e66a96b2d7734ac7b453a733f638317f P a1e61523dc02493e913aaaf41a46c44b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62c6ee18735e40acb8a0a198533d83e0 in term 1.
I20260927 14:57:08.311029 4748 raft_consensus.cc:2454] T e66a96b2d7734ac7b453a733f638317f P 80c231f81022425692df97e6f3e1fbee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62c6ee18735e40acb8a0a198533d83e0 in term 1.
I20260927 14:57:08.311038 4817 leader_election.cc:302] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c6ee18735e40acb8a0a198533d83e0, a1e61523dc02493e913aaaf41a46c44b; no voters:
I20260927 14:57:08.311343 4951 raft_consensus.cc:2781] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:57:08.311553 4951 raft_consensus.cc:686] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0 [term 1 LEADER]: Becoming Leader. State: Replica: 62c6ee18735e40acb8a0a198533d83e0, State: Running, Role: LEADER
I20260927 14:57:08.311883 4951 consensus_queue.cc:237] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e61523dc02493e913aaaf41a46c44b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41595 } } peers { permanent_uuid: "80c231f81022425692df97e6f3e1fbee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33799 } } peers { permanent_uuid: "62c6ee18735e40acb8a0a198533d83e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34077 } }
I20260927 14:57:08.315285 4470 catalog_manager.cc:5495] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62c6ee18735e40acb8a0a198533d83e0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "62c6ee18735e40acb8a0a198533d83e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e61523dc02493e913aaaf41a46c44b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80c231f81022425692df97e6f3e1fbee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62c6ee18735e40acb8a0a198533d83e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34077 } health_report { overall_health: HEALTHY } } }
I20260927 14:57:08.330142 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:08.326905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40260 (local address 127.0.98.3:34077)
0927 14:57:08.327015 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:08.327018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:08.327031 (+ 13us) server_negotiation.cc:407] Connection header received
0927 14:57:08.327118 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:08.327120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:08.327174 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:08.327257 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:57:08.327478 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:08.327979 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:08.328812 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:08.329033 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:08.329112 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:08.329334 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:57:08.329762 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:57:08.329875 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:08.329921 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 14:57:08.329973 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 14:57:08.336097 4615 raft_consensus.cc:1261] T e66a96b2d7734ac7b453a733f638317f P a1e61523dc02493e913aaaf41a46c44b [term 1 FOLLOWER]: Refusing update from remote peer 62c6ee18735e40acb8a0a198533d83e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:57:08.336126 4748 raft_consensus.cc:1261] T e66a96b2d7734ac7b453a733f638317f P 80c231f81022425692df97e6f3e1fbee [term 1 FOLLOWER]: Refusing update from remote peer 62c6ee18735e40acb8a0a198533d83e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:57:08.336809 4954 consensus_queue.cc:1035] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80c231f81022425692df97e6f3e1fbee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:57:08.336998 4951 consensus_queue.cc:1035] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1e61523dc02493e913aaaf41a46c44b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:57:08.343888 4957 mvcc.cc:205] Tried to move back new op lower bound from 7333974132057206784 to 7333974131968200704. Current Snapshot: MvccSnapshot[applied={T|T < 7333974132057206784}]
I20260927 14:57:08.344358 4958 mvcc.cc:205] Tried to move back new op lower bound from 7333974132057206784 to 7333974131968200704. Current Snapshot: MvccSnapshot[applied={T|T < 7333974132057206784}]
I20260927 14:57:08.345884 4959 mvcc.cc:205] Tried to move back new op lower bound from 7333974132057206784 to 7333974131968200704. Current Snapshot: MvccSnapshot[applied={T|T < 7333974132057206784}]
I20260927 14:57:08.358327 4470 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:57:08.358518 4470 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:57:08.361238 4470 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a837f460276c45f682f7f5bf8ade0dc6: Sending DeleteTablet for 3 replicas of tablet e66a96b2d7734ac7b453a733f638317f
I20260927 14:57:08.362140 4728 tablet_service.cc:1514] Processing DeleteTablet for tablet e66a96b2d7734ac7b453a733f638317f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:57:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55718
I20260927 14:57:08.362156 4861 tablet_service.cc:1514] Processing DeleteTablet for tablet e66a96b2d7734ac7b453a733f638317f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:57:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40256
I20260927 14:57:08.362368 4595 tablet_service.cc:1514] Processing DeleteTablet for tablet e66a96b2d7734ac7b453a733f638317f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:57:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45822
I20260927 14:57:08.362906 4968 tablet_replica.cc:330] T e66a96b2d7734ac7b453a733f638317f P a1e61523dc02493e913aaaf41a46c44b: stopping tablet replica
I20260927 14:57:08.363193 4967 tablet_replica.cc:330] T e66a96b2d7734ac7b453a733f638317f P 80c231f81022425692df97e6f3e1fbee: stopping tablet replica
I20260927 14:57:08.363286 4968 raft_consensus.cc:2229] T e66a96b2d7734ac7b453a733f638317f P a1e61523dc02493e913aaaf41a46c44b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:57:08.363297 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 4533
I20260927 14:57:08.364566 4967 raft_consensus.cc:2229] T e66a96b2d7734ac7b453a733f638317f P 80c231f81022425692df97e6f3e1fbee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:57:08.364868 4967 raft_consensus.cc:2258] T e66a96b2d7734ac7b453a733f638317f P 80c231f81022425692df97e6f3e1fbee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:57:08.365206 4966 tablet_replica.cc:330] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0: stopping tablet replica
I20260927 14:57:08.365437 4966 raft_consensus.cc:2229] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:57:08.365818 4966 raft_consensus.cc:2258] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:57:08.367218 4967 ts_tablet_manager.cc:1905] T e66a96b2d7734ac7b453a733f638317f P 80c231f81022425692df97e6f3e1fbee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:57:08.369403 4967 ts_tablet_manager.cc:1918] T e66a96b2d7734ac7b453a733f638317f P 80c231f81022425692df97e6f3e1fbee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:57:08.369481 4967 log.cc:1196] T e66a96b2d7734ac7b453a733f638317f P 80c231f81022425692df97e6f3e1fbee: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-1/wal/wals/e66a96b2d7734ac7b453a733f638317f
I20260927 14:57:08.369699 4967 ts_tablet_manager.cc:1939] T e66a96b2d7734ac7b453a733f638317f P 80c231f81022425692df97e6f3e1fbee: Deleting consensus metadata
I20260927 14:57:08.370965 4457 catalog_manager.cc:4841] TS 80c231f81022425692df97e6f3e1fbee (127.0.98.2:33799): tablet e66a96b2d7734ac7b453a733f638317f (table test-table [id=feb213108f014ff4976913130de2e6a0]) successfully deleted
W20260927 14:57:08.371666 4458 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41595 (error 108)
I20260927 14:57:08.371872 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 4666
I20260927 14:57:08.373378 4966 ts_tablet_manager.cc:1905] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:57:08.376176 4966 ts_tablet_manager.cc:1918] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:57:08.376284 4966 log.cc:1196] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790521004237705-392-0/minicluster-data/ts-2/wal/wals/e66a96b2d7734ac7b453a733f638317f
I20260927 14:57:08.376592 4966 ts_tablet_manager.cc:1939] T e66a96b2d7734ac7b453a733f638317f P 62c6ee18735e40acb8a0a198533d83e0: Deleting consensus metadata
I20260927 14:57:08.377933 4458 catalog_manager.cc:4841] TS 62c6ee18735e40acb8a0a198533d83e0 (127.0.98.3:34077): tablet e66a96b2d7734ac7b453a733f638317f (table test-table [id=feb213108f014ff4976913130de2e6a0]) successfully deleted
I20260927 14:57:08.378592 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:08.378274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41595 (local address 127.0.0.1:45826)
0927 14:57:08.378436 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:08.378503 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260927 14:57:08.378875 4458 catalog_manager.cc:4584] TS a1e61523dc02493e913aaaf41a46c44b (127.0.98.1:41595): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e66a96b2d7734ac7b453a733f638317f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41595: connect: Connection refused (error 111)
I20260927 14:57:08.378942 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 4799
I20260927 14:57:08.383982 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 4440
2026-09-27T14:57:08Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2172 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T14:57:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:57:08Z Disabled control of system clock
I20260927 14:57:08.413978 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35931
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 14:57:08.530102 4973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:08.530390 4973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:08.530428 4973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:08.534559 4973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:57:08.534641 4973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:08.534655 4973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:08.534672 4973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:57:08.534686 4973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:57:08.539759 4973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35931
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:08.540831 4973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:08.542157 4973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:08.551054 4979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:08.551052 4981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:08.551070 4978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:08.551502 4973 server_base.cc:1034] running on GCE node
I20260927 14:57:08.552066 4973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:08.553262 4973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:08.554457 4973 hybrid_clock.cc:648] HybridClock initialized: now 1790521028554430 us; error 44 us; skew 500 ppm
I20260927 14:57:08.556598 4973 webserver.cc:460] Webserver started at http://127.0.98.62:34245/ using document root <none> and password file <none>
I20260927 14:57:08.557204 4973 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:08.557273 4973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:08.557503 4973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:08.559746 4973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "fc1521acc1b34f47b6950d5682d1c749"
format_stamp: "Formatted at 2026-09-27 14:57:08 on dist-test-slave-pz0h"
server_key: "9fdd6f427fa7f0a488c8e6f9c69117fe"
server_key_iv: "6a154c84c99e1d22e18b82ebfa3350ec"
server_key_version: "encryptionkey@0"
I20260927 14:57:08.560334 4973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "fc1521acc1b34f47b6950d5682d1c749"
format_stamp: "Formatted at 2026-09-27 14:57:08 on dist-test-slave-pz0h"
server_key: "9fdd6f427fa7f0a488c8e6f9c69117fe"
server_key_iv: "6a154c84c99e1d22e18b82ebfa3350ec"
server_key_version: "encryptionkey@0"
I20260927 14:57:08.564291 4973 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 14:57:08.566958 4987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:08.568825 4973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:57:08.569053 4973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "fc1521acc1b34f47b6950d5682d1c749"
format_stamp: "Formatted at 2026-09-27 14:57:08 on dist-test-slave-pz0h"
server_key: "9fdd6f427fa7f0a488c8e6f9c69117fe"
server_key_iv: "6a154c84c99e1d22e18b82ebfa3350ec"
server_key_version: "encryptionkey@0"
I20260927 14:57:08.569223 4973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:08.574426 4973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:08.575142 4973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:08.575306 4973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:08.582031 4973 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35931
I20260927 14:57:08.582046 5039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35931 every 8 connection(s)
I20260927 14:57:08.583212 4973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:57:08.586520 5040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:08.590775 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 4973
I20260927 14:57:08.590942 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 14:57:08.591194 392 external_mini_cluster.cc:1428] Setting key b5f74568558dda8ea2e2ccd3ecbb3dd4
I20260927 14:57:08.592489 5040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc1521acc1b34f47b6950d5682d1c749: Bootstrap starting.
I20260927 14:57:08.595360 5040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc1521acc1b34f47b6950d5682d1c749: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:08.596302 5040 log.cc:824] T 00000000000000000000000000000000 P fc1521acc1b34f47b6950d5682d1c749: Log is configured to *not* fsync() on all Append() calls
I20260927 14:57:08.597368 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:08.592580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51782 (local address 127.0.98.62:35931)
0927 14:57:08.593341 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:08.593361 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:08.593417 (+ 56us) server_negotiation.cc:407] Connection header received
0927 14:57:08.594307 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:08.594332 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:08.594842 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:08.595227 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:08.595953 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:08.595959 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:08.595973 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:57:08.595983 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:08.596050 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:57:08.596075 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:08.596188 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:08.596255 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 14:57:08.596523 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":150,"threads_started":1}
I20260927 14:57:08.598444 5040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc1521acc1b34f47b6950d5682d1c749: No bootstrap required, opened a new log
I20260927 14:57:08.602108 5040 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc1521acc1b34f47b6950d5682d1c749 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1521acc1b34f47b6950d5682d1c749" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35931 } }
I20260927 14:57:08.602393 5040 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc1521acc1b34f47b6950d5682d1c749 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:08.602447 5040 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc1521acc1b34f47b6950d5682d1c749 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc1521acc1b34f47b6950d5682d1c749, State: Initialized, Role: FOLLOWER
I20260927 14:57:08.602958 5040 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc1521acc1b34f47b6950d5682d1c749 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1521acc1b34f47b6950d5682d1c749" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35931 } }
I20260927 14:57:08.603104 5040 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc1521acc1b34f47b6950d5682d1c749 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:57:08.603145 5040 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc1521acc1b34f47b6950d5682d1c749 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:57:08.603217 5040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc1521acc1b34f47b6950d5682d1c749 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:08.604136 5040 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc1521acc1b34f47b6950d5682d1c749 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1521acc1b34f47b6950d5682d1c749" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35931 } }
I20260927 14:57:08.604460 5040 leader_election.cc:302] T 00000000000000000000000000000000 P fc1521acc1b34f47b6950d5682d1c749 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc1521acc1b34f47b6950d5682d1c749; no voters:
I20260927 14:57:08.604739 5040 leader_election.cc:288] T 00000000000000000000000000000000 P fc1521acc1b34f47b6950d5682d1c749 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:57:08.604884 5045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc1521acc1b34f47b6950d5682d1c749 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:57:08.605108 5045 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc1521acc1b34f47b6950d5682d1c749 [term 1 LEADER]: Becoming Leader. State: Replica: fc1521acc1b34f47b6950d5682d1c749, State: Running, Role: LEADER
I20260927 14:57:08.605429 5045 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc1521acc1b34f47b6950d5682d1c749 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1521acc1b34f47b6950d5682d1c749" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35931 } }
I20260927 14:57:08.606048 5040 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc1521acc1b34f47b6950d5682d1c749 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:57:08.607048 5047 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc1521acc1b34f47b6950d5682d1c749 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc1521acc1b34f47b6950d5682d1c749. Latest consensus state: current_term: 1 leader_uuid: "fc1521acc1b34f47b6950d5682d1c749" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1521acc1b34f47b6950d5682d1c749" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35931 } } }
I20260927 14:57:08.607144 5047 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc1521acc1b34f47b6950d5682d1c749 [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:08.606997 5046 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc1521acc1b34f47b6950d5682d1c749 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc1521acc1b34f47b6950d5682d1c749" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1521acc1b34f47b6950d5682d1c749" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35931 } } }
I20260927 14:57:08.607271 5046 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc1521acc1b34f47b6950d5682d1c749 [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:08.607533 5054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:57:08.610818 5054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:57:08.617275 5054 catalog_manager.cc:1347] Generated new cluster ID: 8187f967823f41a597553a37d03e2b8d
I20260927 14:57:08.617350 5054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:57:08.635859 5054 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:57:08.636252 5054 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:57:08.647557 5054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc1521acc1b34f47b6950d5682d1c749: Generated new TSK 0
I20260927 14:57:08.648361 5054 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:57:08.656985 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35931
--builtin_ntp_servers=127.0.98.20:39691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 14:57:08.779336 5064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:08.779630 5064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:08.779673 5064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:08.783777 5064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:08.783840 5064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:08.783898 5064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:57:08.788789 5064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:08.789901 5064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:08.791330 5064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:08.801378 5069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:08.801422 5070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:08.801378 5072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:08.801779 5064 server_base.cc:1034] running on GCE node
I20260927 14:57:08.802245 5064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:08.802983 5064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:08.804177 5064 hybrid_clock.cc:648] HybridClock initialized: now 1790521028804157 us; error 39 us; skew 500 ppm
I20260927 14:57:08.809782 5064 webserver.cc:460] Webserver started at http://127.0.98.1:35851/ using document root <none> and password file <none>
I20260927 14:57:08.810480 5064 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:08.810566 5064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:08.810776 5064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:08.813046 5064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-0/data/instance:
uuid: "bdd364cfcf524709a911170f8eab9787"
format_stamp: "Formatted at 2026-09-27 14:57:08 on dist-test-slave-pz0h"
server_key: "1e4aea456024e6c4864d958ed36bf981"
server_key_iv: "1f0b3f131788b77ebc72a01936992f78"
server_key_version: "encryptionkey@0"
I20260927 14:57:08.813647 5064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bdd364cfcf524709a911170f8eab9787"
format_stamp: "Formatted at 2026-09-27 14:57:08 on dist-test-slave-pz0h"
server_key: "1e4aea456024e6c4864d958ed36bf981"
server_key_iv: "1f0b3f131788b77ebc72a01936992f78"
server_key_version: "encryptionkey@0"
I20260927 14:57:08.817726 5064 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 14:57:08.821063 5078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:08.822660 5064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:57:08.822873 5064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "bdd364cfcf524709a911170f8eab9787"
format_stamp: "Formatted at 2026-09-27 14:57:08 on dist-test-slave-pz0h"
server_key: "1e4aea456024e6c4864d958ed36bf981"
server_key_iv: "1f0b3f131788b77ebc72a01936992f78"
server_key_version: "encryptionkey@0"
I20260927 14:57:08.823011 5064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:08.828213 5064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:08.828907 5064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:08.829097 5064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:08.829710 5064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:08.830713 5064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:08.830763 5064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:08.830868 5064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:08.830894 5064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:08.839700 5064 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37145
I20260927 14:57:08.839810 5191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37145 every 8 connection(s)
I20260927 14:57:08.840797 5064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:57:08.844549 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 5064
I20260927 14:57:08.844678 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:57:08.844924 392 external_mini_cluster.cc:1428] Setting key 3460c06f4a0ecceeac67bfa4f941d3ab
I20260927 14:57:08.846886 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35931
--builtin_ntp_servers=127.0.98.20:39691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 14:57:08.847616 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:08.842948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53459 (local address 127.0.98.62:35931)
0927 14:57:08.843123 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:08.843127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:08.844262 (+ 1135us) server_negotiation.cc:407] Connection header received
0927 14:57:08.845767 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:08.845771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:08.845815 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:08.845885 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:08.847140 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:08.847144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:08.847147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:57:08.847160 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:08.847183 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:57:08.847193 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:08.847294 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:08.847425 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 14:57:08.847473 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 14:57:08.848645 5194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:08.843329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35931 (local address 127.0.98.1:53459)
0927 14:57:08.844091 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:08.844135 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:08.845534 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:08.846090 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:08.846100 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:08.846230 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:08.846846 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:57:08.846848 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:08.846898 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:57:08.846978 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:08.847285 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:08.847292 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:08.847297 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 14:57:08.847474 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 14:57:08.847753 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":129,"threads_started":1}
I20260927 14:57:08.850065 5192 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35931
I20260927 14:57:08.850351 5192 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:08.850965 5192 heartbeater.cc:511] Master 127.0.98.62:35931 requested a full tablet report, sending...
I20260927 14:57:08.853006 5004 ts_manager.cc:194] Registered new tserver with Master: bdd364cfcf524709a911170f8eab9787 (127.0.98.1:37145)
W20260927 14:57:08.974044 5195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:08.974327 5195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:08.974367 5195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:08.978469 5195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:08.978543 5195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:08.978602 5195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:57:08.983635 5195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:08.984757 5195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:08.986217 5195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:08.996194 5203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:08.996168 5201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:08.996168 5200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:08.996232 5195 server_base.cc:1034] running on GCE node
I20260927 14:57:08.996922 5195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:08.997611 5195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:08.998845 5195 hybrid_clock.cc:648] HybridClock initialized: now 1790521028998810 us; error 43 us; skew 500 ppm
I20260927 14:57:09.000763 5195 webserver.cc:460] Webserver started at http://127.0.98.2:33859/ using document root <none> and password file <none>
I20260927 14:57:09.001348 5195 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:09.001415 5195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:09.001621 5195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:09.003818 5195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-1/data/instance:
uuid: "78842d0db0444d78ad49de975980707e"
format_stamp: "Formatted at 2026-09-27 14:57:08 on dist-test-slave-pz0h"
server_key: "60389861b21f5560d3a283af3aafb9c7"
server_key_iv: "73227f50eb42ee01ae01488d853c1a27"
server_key_version: "encryptionkey@0"
I20260927 14:57:09.004391 5195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "78842d0db0444d78ad49de975980707e"
format_stamp: "Formatted at 2026-09-27 14:57:08 on dist-test-slave-pz0h"
server_key: "60389861b21f5560d3a283af3aafb9c7"
server_key_iv: "73227f50eb42ee01ae01488d853c1a27"
server_key_version: "encryptionkey@0"
I20260927 14:57:09.008586 5195 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 14:57:09.011519 5209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:09.012835 5195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:57:09.012954 5195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "78842d0db0444d78ad49de975980707e"
format_stamp: "Formatted at 2026-09-27 14:57:08 on dist-test-slave-pz0h"
server_key: "60389861b21f5560d3a283af3aafb9c7"
server_key_iv: "73227f50eb42ee01ae01488d853c1a27"
server_key_version: "encryptionkey@0"
I20260927 14:57:09.013077 5195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:09.017741 5195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:09.018383 5195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:09.018554 5195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:09.019177 5195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:09.020108 5195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:09.020166 5195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:09.020223 5195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:09.020254 5195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:09.028681 5195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44707
I20260927 14:57:09.028719 5322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44707 every 8 connection(s)
I20260927 14:57:09.029692 5195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:57:09.033994 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 5195
I20260927 14:57:09.034118 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:57:09.034408 392 external_mini_cluster.cc:1428] Setting key 4a12b24b98357f4af988a985108593ed
I20260927 14:57:09.036235 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:09.031520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50737 (local address 127.0.98.62:35931)
0927 14:57:09.031688 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:09.031693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:09.032925 (+ 1232us) server_negotiation.cc:407] Connection header received
0927 14:57:09.034363 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:09.034367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:09.034421 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:09.034531 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:09.035786 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:09.035790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:09.035792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:57:09.035805 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:09.035829 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:57:09.035839 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:09.035918 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:09.036053 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 14:57:09.036099 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 14:57:09.036516 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35931
--builtin_ntp_servers=127.0.98.20:39691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 14:57:09.037189 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:09.031962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35931 (local address 127.0.98.2:50737)
0927 14:57:09.032749 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:09.032798 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:09.034029 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:09.034733 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:09.034741 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:09.034903 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:09.035500 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:57:09.035502 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:09.035550 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:57:09.035633 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:09.035914 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:09.035921 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:09.035926 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 14:57:09.036076 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 14:57:09.036305 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":123,"threads_started":1}
I20260927 14:57:09.038443 5323 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35931
I20260927 14:57:09.038686 5323 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:09.039155 5323 heartbeater.cc:511] Master 127.0.98.62:35931 requested a full tablet report, sending...
I20260927 14:57:09.040256 5004 ts_manager.cc:194] Registered new tserver with Master: 78842d0db0444d78ad49de975980707e (127.0.98.2:44707)
W20260927 14:57:09.148355 5326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:09.148626 5326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:09.148667 5326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:09.152915 5326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:09.152985 5326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:09.153043 5326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:57:09.157938 5326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:09.159041 5326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:09.160415 5326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 14:57:09.170120 5326 server_base.cc:1034] running on GCE node
W20260927 14:57:09.170100 5334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:09.170288 5332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:09.170097 5331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:09.170766 5326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:09.171445 5326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:09.172646 5326 hybrid_clock.cc:648] HybridClock initialized: now 1790521029172611 us; error 37 us; skew 500 ppm
I20260927 14:57:09.174724 5326 webserver.cc:460] Webserver started at http://127.0.98.3:34065/ using document root <none> and password file <none>
I20260927 14:57:09.175360 5326 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:09.175428 5326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:09.175637 5326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:09.177769 5326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-2/data/instance:
uuid: "efb6ffe05ef442b4aeae16b77d96361f"
format_stamp: "Formatted at 2026-09-27 14:57:09 on dist-test-slave-pz0h"
server_key: "8e4b6f9348ed5584028a089c58b8a563"
server_key_iv: "cce456a229258238c027b8e9b4b600f3"
server_key_version: "encryptionkey@0"
I20260927 14:57:09.178339 5326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "efb6ffe05ef442b4aeae16b77d96361f"
format_stamp: "Formatted at 2026-09-27 14:57:09 on dist-test-slave-pz0h"
server_key: "8e4b6f9348ed5584028a089c58b8a563"
server_key_iv: "cce456a229258238c027b8e9b4b600f3"
server_key_version: "encryptionkey@0"
I20260927 14:57:09.182272 5326 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 14:57:09.185110 5340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:09.186467 5326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:57:09.186614 5326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "efb6ffe05ef442b4aeae16b77d96361f"
format_stamp: "Formatted at 2026-09-27 14:57:09 on dist-test-slave-pz0h"
server_key: "8e4b6f9348ed5584028a089c58b8a563"
server_key_iv: "cce456a229258238c027b8e9b4b600f3"
server_key_version: "encryptionkey@0"
I20260927 14:57:09.186738 5326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:09.191625 5326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:09.192375 5326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:09.192557 5326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:09.193133 5326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:09.194048 5326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:09.194093 5326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:09.194131 5326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:09.194152 5326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:09.202366 5326 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35477
I20260927 14:57:09.202461 5453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35477 every 8 connection(s)
I20260927 14:57:09.203325 5326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:57:09.209556 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:09.205155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56535 (local address 127.0.98.62:35931)
0927 14:57:09.205316 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:09.205319 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:09.206451 (+ 1132us) server_negotiation.cc:407] Connection header received
0927 14:57:09.207878 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:09.207882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:09.207954 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:09.208031 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:09.209110 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:09.209114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:09.209116 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:57:09.209129 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:09.209149 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:57:09.209158 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:09.209238 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:09.209364 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 14:57:09.209409 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 14:57:09.210431 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:09.205563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35931 (local address 127.0.98.3:56535)
0927 14:57:09.206241 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:09.206351 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:09.207666 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:09.208218 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:09.208225 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:09.208320 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:09.208837 (+ 517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:57:09.208838 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:09.208878 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:57:09.208949 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:09.209230 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:09.209237 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:09.209243 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 14:57:09.209400 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 14:57:09.209647 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":121,"threads_started":1}
I20260927 14:57:09.211544 5454 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35931
I20260927 14:57:09.211755 5454 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:09.212143 5454 heartbeater.cc:511] Master 127.0.98.62:35931 requested a full tablet report, sending...
I20260927 14:57:09.212687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 5326
I20260927 14:57:09.212793 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:57:09.213043 392 external_mini_cluster.cc:1428] Setting key a46145b962c77fae28a022b672928f49
I20260927 14:57:09.213152 5004 ts_manager.cc:194] Registered new tserver with Master: efb6ffe05ef442b4aeae16b77d96361f (127.0.98.3:35477)
I20260927 14:57:09.214869 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:57:09.218117 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:09.216672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45658 (local address 127.0.98.62:35931)
0927 14:57:09.216814 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:09.216818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:09.216935 (+ 117us) server_negotiation.cc:407] Connection header received
0927 14:57:09.217043 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:09.217045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:09.217078 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:09.217162 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:09.217802 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:09.217804 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:09.217807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:57:09.217815 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:09.217831 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:57:09.217838 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:09.217896 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:09.217950 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 14:57:09.217983 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 14:57:09.220947 5004 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:57:09.223973 5004 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:57:09.235517 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:09.232743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37145 (local address 127.0.0.1:59296)
0927 14:57:09.233186 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:09.233197 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:09.233265 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:09.234120 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:09.234122 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:09.234133 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:09.234687 (+ 554us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:57:09.234689 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:09.234710 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:57:09.234785 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:09.235165 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:09.235174 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:09.235180 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 14:57:09.235298 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 14:57:09.235367 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":95,"threads_started":1}
I20260927 14:57:09.235666 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:09.233269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59296 (local address 127.0.98.1:37145)
0927 14:57:09.233672 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:09.233684 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:09.233707 (+ 23us) server_negotiation.cc:407] Connection header received
0927 14:57:09.233758 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:09.233763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:09.233939 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:09.234133 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:09.234960 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:09.234967 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:09.234979 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:57:09.234991 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:09.235042 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:57:09.235072 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:09.235173 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:09.235317 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 14:57:09.235404 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":128,"threads_started":1}
I20260927 14:57:09.235744 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:09.233147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35477 (local address 127.0.0.1:57888)
0927 14:57:09.233363 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:09.233377 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:09.233451 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:09.234319 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:09.234322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:09.234337 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:09.234820 (+ 483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:57:09.234822 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:09.234835 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:57:09.234862 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:09.235548 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:09.235551 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:09.235555 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 14:57:09.235609 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 14:57:09.235640 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":88,"threads_started":1}
I20260927 14:57:09.236073 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:09.232386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44707 (local address 127.0.0.1:55932)
0927 14:57:09.232804 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:09.232870 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:09.232979 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:09.233997 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:09.234004 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:09.234075 (+ 71us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:09.235012 (+ 937us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:57:09.235014 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:09.235023 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:57:09.235044 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:09.235865 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:09.235869 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:09.235872 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 14:57:09.235927 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 14:57:09.235959 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":116,"threads_started":1}
I20260927 14:57:09.236259 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:09.233558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57888 (local address 127.0.98.3:35477)
0927 14:57:09.233939 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:09.233950 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:09.233971 (+ 21us) server_negotiation.cc:407] Connection header received
0927 14:57:09.234022 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:09.234027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:09.234184 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:09.234572 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:09.235362 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:09.235367 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:09.235377 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:57:09.235386 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:09.235420 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:57:09.235440 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:09.235991 (+ 551us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:09.236037 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 14:57:09.236105 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":115,"threads_started":1}
I20260927 14:57:09.236589 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:09.232504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55932 (local address 127.0.98.2:44707)
0927 14:57:09.233517 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:09.233528 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:09.233554 (+ 26us) server_negotiation.cc:407] Connection header received
0927 14:57:09.233608 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:09.233614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:09.233822 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:09.234925 (+ 1103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:09.235667 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:09.235672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:09.235682 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:57:09.235692 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:09.235727 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:57:09.235750 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:09.236325 (+ 575us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:09.236370 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 14:57:09.236432 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":903,"thread_start_us":126,"threads_started":1}
I20260927 14:57:09.238561 5126 tablet_service.cc:1467] Processing CreateTablet for tablet 3e6a78499e2f4fc8a3a895b5234ba7fe (DEFAULT_TABLE table=test-table [id=4ba2a9b449714beaa4c5d16f49c5ec42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:57:09.238752 5257 tablet_service.cc:1467] Processing CreateTablet for tablet 3e6a78499e2f4fc8a3a895b5234ba7fe (DEFAULT_TABLE table=test-table [id=4ba2a9b449714beaa4c5d16f49c5ec42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:57:09.239537 5126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e6a78499e2f4fc8a3a895b5234ba7fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:09.239583 5257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e6a78499e2f4fc8a3a895b5234ba7fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:09.242532 5388 tablet_service.cc:1467] Processing CreateTablet for tablet 3e6a78499e2f4fc8a3a895b5234ba7fe (DEFAULT_TABLE table=test-table [id=4ba2a9b449714beaa4c5d16f49c5ec42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:57:09.243723 5388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e6a78499e2f4fc8a3a895b5234ba7fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:09.245843 5471 tablet_bootstrap.cc:492] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e: Bootstrap starting.
I20260927 14:57:09.247272 5472 tablet_bootstrap.cc:492] T 3e6a78499e2f4fc8a3a895b5234ba7fe P bdd364cfcf524709a911170f8eab9787: Bootstrap starting.
I20260927 14:57:09.248265 5471 tablet_bootstrap.cc:654] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:09.249142 5471 log.cc:824] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e: Log is configured to *not* fsync() on all Append() calls
I20260927 14:57:09.249394 5472 tablet_bootstrap.cc:654] T 3e6a78499e2f4fc8a3a895b5234ba7fe P bdd364cfcf524709a911170f8eab9787: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:09.250168 5472 log.cc:824] T 3e6a78499e2f4fc8a3a895b5234ba7fe P bdd364cfcf524709a911170f8eab9787: Log is configured to *not* fsync() on all Append() calls
I20260927 14:57:09.250495 5474 tablet_bootstrap.cc:492] T 3e6a78499e2f4fc8a3a895b5234ba7fe P efb6ffe05ef442b4aeae16b77d96361f: Bootstrap starting.
I20260927 14:57:09.251488 5471 tablet_bootstrap.cc:492] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e: No bootstrap required, opened a new log
I20260927 14:57:09.251689 5471 ts_tablet_manager.cc:1397] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 14:57:09.252384 5472 tablet_bootstrap.cc:492] T 3e6a78499e2f4fc8a3a895b5234ba7fe P bdd364cfcf524709a911170f8eab9787: No bootstrap required, opened a new log
I20260927 14:57:09.252410 5474 tablet_bootstrap.cc:654] T 3e6a78499e2f4fc8a3a895b5234ba7fe P efb6ffe05ef442b4aeae16b77d96361f: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:09.252580 5472 ts_tablet_manager.cc:1397] T 3e6a78499e2f4fc8a3a895b5234ba7fe P bdd364cfcf524709a911170f8eab9787: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 14:57:09.253082 5474 log.cc:824] T 3e6a78499e2f4fc8a3a895b5234ba7fe P efb6ffe05ef442b4aeae16b77d96361f: Log is configured to *not* fsync() on all Append() calls
I20260927 14:57:09.254727 5474 tablet_bootstrap.cc:492] T 3e6a78499e2f4fc8a3a895b5234ba7fe P efb6ffe05ef442b4aeae16b77d96361f: No bootstrap required, opened a new log
I20260927 14:57:09.254951 5474 ts_tablet_manager.cc:1397] T 3e6a78499e2f4fc8a3a895b5234ba7fe P efb6ffe05ef442b4aeae16b77d96361f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 14:57:09.256326 5471 raft_consensus.cc:348] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78842d0db0444d78ad49de975980707e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "efb6ffe05ef442b4aeae16b77d96361f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } peers { permanent_uuid: "bdd364cfcf524709a911170f8eab9787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37145 } }
I20260927 14:57:09.256630 5471 raft_consensus.cc:374] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:09.256690 5471 raft_consensus.cc:729] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78842d0db0444d78ad49de975980707e, State: Initialized, Role: FOLLOWER
I20260927 14:57:09.256881 5472 raft_consensus.cc:348] T 3e6a78499e2f4fc8a3a895b5234ba7fe P bdd364cfcf524709a911170f8eab9787 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78842d0db0444d78ad49de975980707e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "efb6ffe05ef442b4aeae16b77d96361f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } peers { permanent_uuid: "bdd364cfcf524709a911170f8eab9787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37145 } }
I20260927 14:57:09.257117 5472 raft_consensus.cc:374] T 3e6a78499e2f4fc8a3a895b5234ba7fe P bdd364cfcf524709a911170f8eab9787 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:09.257174 5472 raft_consensus.cc:729] T 3e6a78499e2f4fc8a3a895b5234ba7fe P bdd364cfcf524709a911170f8eab9787 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdd364cfcf524709a911170f8eab9787, State: Initialized, Role: FOLLOWER
I20260927 14:57:09.257159 5471 consensus_queue.cc:260] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78842d0db0444d78ad49de975980707e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "efb6ffe05ef442b4aeae16b77d96361f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } peers { permanent_uuid: "bdd364cfcf524709a911170f8eab9787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37145 } }
I20260927 14:57:09.257637 5472 consensus_queue.cc:260] T 3e6a78499e2f4fc8a3a895b5234ba7fe P bdd364cfcf524709a911170f8eab9787 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78842d0db0444d78ad49de975980707e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "efb6ffe05ef442b4aeae16b77d96361f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } peers { permanent_uuid: "bdd364cfcf524709a911170f8eab9787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37145 } }
I20260927 14:57:09.257869 5323 heartbeater.cc:503] Master 127.0.98.62:35931 was elected leader, sending a full tablet report...
I20260927 14:57:09.258293 5192 heartbeater.cc:503] Master 127.0.98.62:35931 was elected leader, sending a full tablet report...
I20260927 14:57:09.258551 5472 ts_tablet_manager.cc:1428] T 3e6a78499e2f4fc8a3a895b5234ba7fe P bdd364cfcf524709a911170f8eab9787: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 14:57:09.258752 5471 ts_tablet_manager.cc:1428] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260927 14:57:09.259861 5474 raft_consensus.cc:348] T 3e6a78499e2f4fc8a3a895b5234ba7fe P efb6ffe05ef442b4aeae16b77d96361f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78842d0db0444d78ad49de975980707e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "efb6ffe05ef442b4aeae16b77d96361f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } peers { permanent_uuid: "bdd364cfcf524709a911170f8eab9787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37145 } }
I20260927 14:57:09.260079 5474 raft_consensus.cc:374] T 3e6a78499e2f4fc8a3a895b5234ba7fe P efb6ffe05ef442b4aeae16b77d96361f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:09.260136 5474 raft_consensus.cc:729] T 3e6a78499e2f4fc8a3a895b5234ba7fe P efb6ffe05ef442b4aeae16b77d96361f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb6ffe05ef442b4aeae16b77d96361f, State: Initialized, Role: FOLLOWER
I20260927 14:57:09.260527 5474 consensus_queue.cc:260] T 3e6a78499e2f4fc8a3a895b5234ba7fe P efb6ffe05ef442b4aeae16b77d96361f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78842d0db0444d78ad49de975980707e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "efb6ffe05ef442b4aeae16b77d96361f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } peers { permanent_uuid: "bdd364cfcf524709a911170f8eab9787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37145 } }
I20260927 14:57:09.261169 5454 heartbeater.cc:503] Master 127.0.98.62:35931 was elected leader, sending a full tablet report...
I20260927 14:57:09.262082 5474 ts_tablet_manager.cc:1428] T 3e6a78499e2f4fc8a3a895b5234ba7fe P efb6ffe05ef442b4aeae16b77d96361f: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
W20260927 14:57:09.280660 5324 tablet.cc:2369] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 14:57:09.342046 5193 tablet.cc:2369] T 3e6a78499e2f4fc8a3a895b5234ba7fe P bdd364cfcf524709a911170f8eab9787: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 14:57:09.454344 5455 tablet.cc:2369] T 3e6a78499e2f4fc8a3a895b5234ba7fe P efb6ffe05ef442b4aeae16b77d96361f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:57:09.564347 5477 raft_consensus.cc:482] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:57:09.564580 5477 raft_consensus.cc:504] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78842d0db0444d78ad49de975980707e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "efb6ffe05ef442b4aeae16b77d96361f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } peers { permanent_uuid: "bdd364cfcf524709a911170f8eab9787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37145 } }
I20260927 14:57:09.565696 5477 leader_election.cc:288] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efb6ffe05ef442b4aeae16b77d96361f (127.0.98.3:35477), bdd364cfcf524709a911170f8eab9787 (127.0.98.1:37145)
I20260927 14:57:09.567816 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:09.566067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60681 (local address 127.0.98.3:35477)
0927 14:57:09.566206 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:09.566211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:09.566428 (+ 217us) server_negotiation.cc:407] Connection header received
0927 14:57:09.566537 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:09.566541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:09.566583 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:09.566646 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:09.567425 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:09.567429 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:09.567432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:57:09.567444 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:09.567466 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:57:09.567476 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:09.567562 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:09.567635 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 14:57:09.567679 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 14:57:09.567842 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:09.565972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35477 (local address 127.0.98.2:60681)
0927 14:57:09.566330 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:09.566342 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:09.566444 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:09.566688 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:09.566690 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:09.566705 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:09.567245 (+ 540us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:57:09.567247 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:09.567260 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:57:09.567292 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:09.567551 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:09.567555 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:09.567557 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 14:57:09.567639 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 14:57:09.567678 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":120,"threads_started":1}
I20260927 14:57:09.567901 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:09.566192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41451 (local address 127.0.98.1:37145)
0927 14:57:09.566334 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:09.566338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:09.566489 (+ 151us) server_negotiation.cc:407] Connection header received
0927 14:57:09.566572 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:09.566575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:09.566609 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:09.566681 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:09.567553 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:09.567556 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:09.567559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:57:09.567571 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:09.567593 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:57:09.567604 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:09.567685 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:09.567754 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 14:57:09.567790 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 14:57:09.568107 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:09.565999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37145 (local address 127.0.98.2:41451)
0927 14:57:09.566374 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:09.566386 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:09.566494 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:09.566760 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:09.566763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:09.566776 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:09.567390 (+ 614us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:57:09.567392 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:09.567403 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:57:09.567428 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:09.567679 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:09.567682 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:09.567685 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 14:57:09.567757 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 14:57:09.567788 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":89,"threads_started":1}
I20260927 14:57:09.568447 5408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6a78499e2f4fc8a3a895b5234ba7fe" candidate_uuid: "78842d0db0444d78ad49de975980707e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb6ffe05ef442b4aeae16b77d96361f" is_pre_election: true
I20260927 14:57:09.568713 5408 raft_consensus.cc:2454] T 3e6a78499e2f4fc8a3a895b5234ba7fe P efb6ffe05ef442b4aeae16b77d96361f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78842d0db0444d78ad49de975980707e in term 0.
I20260927 14:57:09.569140 5212 leader_election.cc:302] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78842d0db0444d78ad49de975980707e, efb6ffe05ef442b4aeae16b77d96361f; no voters:
I20260927 14:57:09.569420 5477 raft_consensus.cc:2781] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:57:09.569429 5146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6a78499e2f4fc8a3a895b5234ba7fe" candidate_uuid: "78842d0db0444d78ad49de975980707e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdd364cfcf524709a911170f8eab9787" is_pre_election: true
I20260927 14:57:09.569504 5477 raft_consensus.cc:482] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:57:09.569555 5477 raft_consensus.cc:3037] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:09.569679 5146 raft_consensus.cc:2454] T 3e6a78499e2f4fc8a3a895b5234ba7fe P bdd364cfcf524709a911170f8eab9787 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78842d0db0444d78ad49de975980707e in term 0.
I20260927 14:57:09.570667 5477 raft_consensus.cc:504] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78842d0db0444d78ad49de975980707e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "efb6ffe05ef442b4aeae16b77d96361f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } peers { permanent_uuid: "bdd364cfcf524709a911170f8eab9787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37145 } }
I20260927 14:57:09.571024 5477 leader_election.cc:288] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e [CANDIDATE]: Term 1 election: Requested vote from peers efb6ffe05ef442b4aeae16b77d96361f (127.0.98.3:35477), bdd364cfcf524709a911170f8eab9787 (127.0.98.1:37145)
I20260927 14:57:09.571334 5146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6a78499e2f4fc8a3a895b5234ba7fe" candidate_uuid: "78842d0db0444d78ad49de975980707e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdd364cfcf524709a911170f8eab9787"
I20260927 14:57:09.571453 5146 raft_consensus.cc:3037] T 3e6a78499e2f4fc8a3a895b5234ba7fe P bdd364cfcf524709a911170f8eab9787 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:09.571480 5408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6a78499e2f4fc8a3a895b5234ba7fe" candidate_uuid: "78842d0db0444d78ad49de975980707e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb6ffe05ef442b4aeae16b77d96361f"
I20260927 14:57:09.571601 5408 raft_consensus.cc:3037] T 3e6a78499e2f4fc8a3a895b5234ba7fe P efb6ffe05ef442b4aeae16b77d96361f [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:09.572479 5146 raft_consensus.cc:2454] T 3e6a78499e2f4fc8a3a895b5234ba7fe P bdd364cfcf524709a911170f8eab9787 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78842d0db0444d78ad49de975980707e in term 1.
I20260927 14:57:09.572558 5408 raft_consensus.cc:2454] T 3e6a78499e2f4fc8a3a895b5234ba7fe P efb6ffe05ef442b4aeae16b77d96361f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78842d0db0444d78ad49de975980707e in term 1.
I20260927 14:57:09.572746 5210 leader_election.cc:302] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78842d0db0444d78ad49de975980707e, bdd364cfcf524709a911170f8eab9787; no voters:
I20260927 14:57:09.572911 5477 raft_consensus.cc:2781] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:57:09.573137 5477 raft_consensus.cc:686] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e [term 1 LEADER]: Becoming Leader. State: Replica: 78842d0db0444d78ad49de975980707e, State: Running, Role: LEADER
I20260927 14:57:09.573441 5477 consensus_queue.cc:237] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78842d0db0444d78ad49de975980707e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } } peers { permanent_uuid: "efb6ffe05ef442b4aeae16b77d96361f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } peers { permanent_uuid: "bdd364cfcf524709a911170f8eab9787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37145 } }
I20260927 14:57:09.576622 5004 catalog_manager.cc:5495] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e reported cstate change: term changed from 0 to 1, leader changed from <none> to 78842d0db0444d78ad49de975980707e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "78842d0db0444d78ad49de975980707e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78842d0db0444d78ad49de975980707e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb6ffe05ef442b4aeae16b77d96361f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdd364cfcf524709a911170f8eab9787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37145 } health_report { overall_health: UNKNOWN } } }
I20260927 14:57:09.599967 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:09.598708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55934 (local address 127.0.98.2:44707)
0927 14:57:09.598850 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:09.598854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:09.598867 (+ 13us) server_negotiation.cc:407] Connection header received
0927 14:57:09.598967 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:09.598970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:09.599033 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:09.599112 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:09.599700 (+ 588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:09.599702 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:09.599704 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:57:09.599711 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:09.599725 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:57:09.599731 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:09.599791 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:09.599828 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 14:57:09.599861 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 14:57:09.605803 5408 raft_consensus.cc:1261] T 3e6a78499e2f4fc8a3a895b5234ba7fe P efb6ffe05ef442b4aeae16b77d96361f [term 1 FOLLOWER]: Refusing update from remote peer 78842d0db0444d78ad49de975980707e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:57:09.605824 5146 raft_consensus.cc:1261] T 3e6a78499e2f4fc8a3a895b5234ba7fe P bdd364cfcf524709a911170f8eab9787 [term 1 FOLLOWER]: Refusing update from remote peer 78842d0db0444d78ad49de975980707e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:57:09.606613 5482 consensus_queue.cc:1035] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdd364cfcf524709a911170f8eab9787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:57:09.606846 5477 consensus_queue.cc:1035] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb6ffe05ef442b4aeae16b77d96361f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:57:09.618286 5487 mvcc.cc:205] Tried to move back new op lower bound from 7333974137258364928 to 7333974137135566848. Current Snapshot: MvccSnapshot[applied={T|T < 7333974137258364928 or (T in {7333974137258364928})}]
I20260927 14:57:09.624110 5004 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:57:09.624275 5004 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:57:09.626907 5004 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc1521acc1b34f47b6950d5682d1c749: Sending DeleteTablet for 3 replicas of tablet 3e6a78499e2f4fc8a3a895b5234ba7fe
I20260927 14:57:09.627837 5126 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e6a78499e2f4fc8a3a895b5234ba7fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:57:09 UTC) from {username='slave'} at 127.0.0.1:59296
I20260927 14:57:09.627879 5388 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e6a78499e2f4fc8a3a895b5234ba7fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:57:09 UTC) from {username='slave'} at 127.0.0.1:57888
I20260927 14:57:09.628295 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 5064
I20260927 14:57:09.630256 5257 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e6a78499e2f4fc8a3a895b5234ba7fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:57:09 UTC) from {username='slave'} at 127.0.0.1:55932
I20260927 14:57:09.630333 5495 tablet_replica.cc:330] T 3e6a78499e2f4fc8a3a895b5234ba7fe P efb6ffe05ef442b4aeae16b77d96361f: stopping tablet replica
I20260927 14:57:09.630728 5497 tablet_replica.cc:330] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e: stopping tablet replica
I20260927 14:57:09.630719 5495 raft_consensus.cc:2229] T 3e6a78499e2f4fc8a3a895b5234ba7fe P efb6ffe05ef442b4aeae16b77d96361f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:57:09.631029 5497 raft_consensus.cc:2229] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:57:09.631132 5495 raft_consensus.cc:2258] T 3e6a78499e2f4fc8a3a895b5234ba7fe P efb6ffe05ef442b4aeae16b77d96361f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:57:09.631455 5497 raft_consensus.cc:2258] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:57:09.632756 5497 ts_tablet_manager.cc:1905] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:57:09.632799 5495 ts_tablet_manager.cc:1905] T 3e6a78499e2f4fc8a3a895b5234ba7fe P efb6ffe05ef442b4aeae16b77d96361f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:57:09.635704 5497 ts_tablet_manager.cc:1918] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:57:09.635743 5495 ts_tablet_manager.cc:1918] T 3e6a78499e2f4fc8a3a895b5234ba7fe P efb6ffe05ef442b4aeae16b77d96361f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:57:09.635831 5497 log.cc:1196] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-1/wal/wals/3e6a78499e2f4fc8a3a895b5234ba7fe
I20260927 14:57:09.635843 5495 log.cc:1196] T 3e6a78499e2f4fc8a3a895b5234ba7fe P efb6ffe05ef442b4aeae16b77d96361f: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790521004237705-392-0/minicluster-data/ts-2/wal/wals/3e6a78499e2f4fc8a3a895b5234ba7fe
I20260927 14:57:09.636138 5495 ts_tablet_manager.cc:1939] T 3e6a78499e2f4fc8a3a895b5234ba7fe P efb6ffe05ef442b4aeae16b77d96361f: Deleting consensus metadata
I20260927 14:57:09.636154 5497 ts_tablet_manager.cc:1939] T 3e6a78499e2f4fc8a3a895b5234ba7fe P 78842d0db0444d78ad49de975980707e: Deleting consensus metadata
I20260927 14:57:09.637530 4990 catalog_manager.cc:4841] TS efb6ffe05ef442b4aeae16b77d96361f (127.0.98.3:35477): tablet 3e6a78499e2f4fc8a3a895b5234ba7fe (table test-table [id=4ba2a9b449714beaa4c5d16f49c5ec42]) successfully deleted
I20260927 14:57:09.637758 4990 catalog_manager.cc:4841] TS 78842d0db0444d78ad49de975980707e (127.0.98.2:44707): tablet 3e6a78499e2f4fc8a3a895b5234ba7fe (table test-table [id=4ba2a9b449714beaa4c5d16f49c5ec42]) successfully deleted
W20260927 14:57:09.639773 4988 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37145 (error 108)
I20260927 14:57:09.639856 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 5195
I20260927 14:57:09.641242 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:09.640902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37145 (local address 127.0.0.1:59304)
0927 14:57:09.641084 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:09.641148 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260927 14:57:09.641428 4988 catalog_manager.cc:4584] TS bdd364cfcf524709a911170f8eab9787 (127.0.98.1:37145): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3e6a78499e2f4fc8a3a895b5234ba7fe: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37145: connect: Connection refused (error 111)
I20260927 14:57:09.645166 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 5326
I20260927 14:57:09.650827 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 4973
2026-09-27T14:57:09Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1265 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 14:57:09.671631 392 mini_oidc.cc:152] Starting JWKS server
I20260927 14:57:09.673254 392 webserver.cc:460] Webserver started at https://127.0.0.1:39535/ using document root <none> and password file <none>
I20260927 14:57:09.673430 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 14:57:09.673710 392 webserver.cc:460] Webserver started at http://127.0.0.1:41259/ using document root <none> and password file <none>
2026-09-27T14:57:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:57:09Z Disabled control of system clock
I20260927 14:57:09.682571 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33697
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33697
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39535/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521004237705-392-0/testchainca.cert with env {}
W20260927 14:57:09.795806 5504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:09.796116 5504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:09.796159 5504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:09.800415 5504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:57:09.800490 5504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:09.800505 5504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:09.800521 5504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:57:09.800535 5504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 14:57:09.800555 5504 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 14:57:09.800582 5504 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39535/jwks.json
I20260927 14:57:09.806572 5504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33697
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39535/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521004237705-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:09.807826 5504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:09.808882 5504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:09.817621 5509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:09.817569 5512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:09.817585 5504 server_base.cc:1034] running on GCE node
W20260927 14:57:09.817598 5510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:09.818436 5504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:09.819720 5504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:09.820899 5504 hybrid_clock.cc:648] HybridClock initialized: now 1790521029820880 us; error 35 us; skew 500 ppm
I20260927 14:57:09.823052 5504 webserver.cc:460] Webserver started at http://127.0.98.62:34237/ using document root <none> and password file <none>
I20260927 14:57:09.823663 5504 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:09.823732 5504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:09.823948 5504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:09.826146 5504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "70e7cd3aa564421ba4a32276428f4585"
format_stamp: "Formatted at 2026-09-27 14:57:09 on dist-test-slave-pz0h"
server_key: "6742cca0e76ba2912ee975dfd53c2967"
server_key_iv: "07fa51f8685a3395c818196b0f2897f6"
server_key_version: "encryptionkey@0"
I20260927 14:57:09.826753 5504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "70e7cd3aa564421ba4a32276428f4585"
format_stamp: "Formatted at 2026-09-27 14:57:09 on dist-test-slave-pz0h"
server_key: "6742cca0e76ba2912ee975dfd53c2967"
server_key_iv: "07fa51f8685a3395c818196b0f2897f6"
server_key_version: "encryptionkey@0"
I20260927 14:57:09.830895 5504 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 14:57:09.833619 5518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:09.835047 5504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:57:09.835201 5504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "70e7cd3aa564421ba4a32276428f4585"
format_stamp: "Formatted at 2026-09-27 14:57:09 on dist-test-slave-pz0h"
server_key: "6742cca0e76ba2912ee975dfd53c2967"
server_key_iv: "07fa51f8685a3395c818196b0f2897f6"
server_key_version: "encryptionkey@0"
I20260927 14:57:09.835319 5504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:09.890122 5504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:09.890952 5504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:09.891149 5504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:09.898007 5504 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33697
I20260927 14:57:09.898025 5570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33697 every 8 connection(s)
I20260927 14:57:09.899051 5504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:57:09.899375 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 5504
I20260927 14:57:09.899472 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 14:57:09.899776 392 external_mini_cluster.cc:1428] Setting key 4d68e68acd4188bb04c35ff5ff16034d
I20260927 14:57:09.902864 5571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:09.910511 5571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e7cd3aa564421ba4a32276428f4585: Bootstrap starting.
I20260927 14:57:09.910912 5574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:09.901145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34458 (local address 127.0.98.62:33697)
0927 14:57:09.901847 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:09.901867 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:09.901911 (+ 44us) server_negotiation.cc:407] Connection header received
0927 14:57:09.902787 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:09.902843 (+ 56us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:09.903372 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:09.903785 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:09.904240 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:09.905753 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:09.906746 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:09.907602 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:09.907770 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:09.908641 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:09.908648 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:09.908667 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:57:09.909225 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:09.909309 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:57:09.909338 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:09.909459 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:09.909591 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 14:57:09.909891 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":124,"threads_started":1}
I20260927 14:57:09.913112 5571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70e7cd3aa564421ba4a32276428f4585: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:09.913906 5571 log.cc:824] T 00000000000000000000000000000000 P 70e7cd3aa564421ba4a32276428f4585: Log is configured to *not* fsync() on all Append() calls
I20260927 14:57:09.913993 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:57:09.916129 5571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e7cd3aa564421ba4a32276428f4585: No bootstrap required, opened a new log
I20260927 14:57:09.919234 5571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70e7cd3aa564421ba4a32276428f4585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e7cd3aa564421ba4a32276428f4585" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } }
I20260927 14:57:09.919507 5571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70e7cd3aa564421ba4a32276428f4585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:09.919574 5571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70e7cd3aa564421ba4a32276428f4585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70e7cd3aa564421ba4a32276428f4585, State: Initialized, Role: FOLLOWER
I20260927 14:57:09.920028 5571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70e7cd3aa564421ba4a32276428f4585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e7cd3aa564421ba4a32276428f4585" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } }
I20260927 14:57:09.920167 5571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70e7cd3aa564421ba4a32276428f4585 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:57:09.920234 5571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70e7cd3aa564421ba4a32276428f4585 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:57:09.920336 5571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70e7cd3aa564421ba4a32276428f4585 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:09.921373 5571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70e7cd3aa564421ba4a32276428f4585 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e7cd3aa564421ba4a32276428f4585" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } }
I20260927 14:57:09.921772 5571 leader_election.cc:302] T 00000000000000000000000000000000 P 70e7cd3aa564421ba4a32276428f4585 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70e7cd3aa564421ba4a32276428f4585; no voters:
I20260927 14:57:09.922151 5571 leader_election.cc:288] T 00000000000000000000000000000000 P 70e7cd3aa564421ba4a32276428f4585 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:57:09.922294 5576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70e7cd3aa564421ba4a32276428f4585 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:57:09.922535 5576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70e7cd3aa564421ba4a32276428f4585 [term 1 LEADER]: Becoming Leader. State: Replica: 70e7cd3aa564421ba4a32276428f4585, State: Running, Role: LEADER
I20260927 14:57:09.922894 5576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70e7cd3aa564421ba4a32276428f4585 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e7cd3aa564421ba4a32276428f4585" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } }
I20260927 14:57:09.923542 5571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70e7cd3aa564421ba4a32276428f4585 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:57:09.924690 5578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70e7cd3aa564421ba4a32276428f4585 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70e7cd3aa564421ba4a32276428f4585. Latest consensus state: current_term: 1 leader_uuid: "70e7cd3aa564421ba4a32276428f4585" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e7cd3aa564421ba4a32276428f4585" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } } }
I20260927 14:57:09.924832 5578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70e7cd3aa564421ba4a32276428f4585 [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:09.925118 5577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70e7cd3aa564421ba4a32276428f4585 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70e7cd3aa564421ba4a32276428f4585" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e7cd3aa564421ba4a32276428f4585" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } } }
I20260927 14:57:09.925222 5577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70e7cd3aa564421ba4a32276428f4585 [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:09.925345 5585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:57:09.929591 5585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:57:09.936214 5585 catalog_manager.cc:1347] Generated new cluster ID: 55683e63fe984f04b4746357d369bba8
I20260927 14:57:09.936319 5585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:57:09.957962 5585 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:57:09.959004 5585 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:57:09.968834 5585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70e7cd3aa564421ba4a32276428f4585: Generated new TSK 0
I20260927 14:57:09.969616 5585 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 14:57:09.999744 5601 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 14:57:09.999884 5601 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 14:57:10.000442 5574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:09.982560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34460 (local address 127.0.98.62:33697)
0927 14:57:09.982703 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:09.982708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:09.982941 (+ 233us) server_negotiation.cc:407] Connection header received
0927 14:57:09.983230 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:09.983234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:09.983293 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:09.983473 (+ 180us) server_negotiation.cc:226] Negotiated authn=JWT
0927 14:57:09.983678 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:09.984216 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:09.985493 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:09.985714 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:09.985796 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:09.986199 (+ 403us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 14:57:10.000054 (+ 13855us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39535/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 14:57:10.000305 (+ 251us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34460: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39535/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 14:57:10.000502 5574 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34460: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39535/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 14:57:10.001998 5600 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:09.982404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33697 (local address 127.0.0.1:34460)
0927 14:57:09.982833 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:09.982850 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:09.983006 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:09.983425 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:09.983428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:09.983436 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 14:57:09.983573 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:09.983580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:09.984321 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:09.984325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:09.985353 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:09.985364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:09.985805 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:09.985837 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:09.985937 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:09.986002 (+ 65us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 14:57:10.000317 (+ 14315us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39535/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 14:57:10.000601 (+ 284us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33697: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39535/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":134,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:33697: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39535/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 14:57:10.002849 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 5504
2026-09-27T14:57:10Z chronyd exiting
I20260927 14:57:10.040201 392 test_util.cc:175] -----------------------------------------------
I20260927 14:57:10.040298 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790521004237705-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (370 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 14:57:10.041373 392 mini_oidc.cc:152] Starting JWKS server
I20260927 14:57:10.042276 392 webserver.cc:460] Webserver started at https://127.0.0.1:41559/ using document root <none> and password file <none>
I20260927 14:57:10.042403 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 14:57:10.042677 392 webserver.cc:460] Webserver started at http://127.0.0.1:38713/ using document root <none> and password file <none>
2026-09-27T14:57:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:57:10Z Disabled control of system clock
I20260927 14:57:10.053064 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35229
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35229
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41559/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521004237705-392-0/testchainca.cert with env {}
W20260927 14:57:10.168594 5607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:10.168896 5607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:10.168967 5607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:10.173244 5607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:57:10.173331 5607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:10.173353 5607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:10.173374 5607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:57:10.173394 5607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 14:57:10.173421 5607 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 14:57:10.173444 5607 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41559/jwks.json
I20260927 14:57:10.179255 5607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35229
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41559/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521004237705-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:10.180421 5607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:10.181471 5607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:10.191067 5615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:10.191067 5613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:10.191067 5612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:10.191524 5607 server_base.cc:1034] running on GCE node
I20260927 14:57:10.192129 5607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:10.193307 5607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:10.194504 5607 hybrid_clock.cc:648] HybridClock initialized: now 1790521030194490 us; error 33 us; skew 500 ppm
I20260927 14:57:10.196532 5607 webserver.cc:460] Webserver started at http://127.0.98.62:39769/ using document root <none> and password file <none>
I20260927 14:57:10.197129 5607 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:10.197201 5607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:10.197402 5607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:10.199628 5607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "413f79d942cf4524906ff800d6b621ea"
format_stamp: "Formatted at 2026-09-27 14:57:10 on dist-test-slave-pz0h"
server_key: "8cfb7aeadcda20b9b9f1156d51b30bff"
server_key_iv: "6aea589b07343b0dfd4fcdadd5aa4ee7"
server_key_version: "encryptionkey@0"
I20260927 14:57:10.200201 5607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "413f79d942cf4524906ff800d6b621ea"
format_stamp: "Formatted at 2026-09-27 14:57:10 on dist-test-slave-pz0h"
server_key: "8cfb7aeadcda20b9b9f1156d51b30bff"
server_key_iv: "6aea589b07343b0dfd4fcdadd5aa4ee7"
server_key_version: "encryptionkey@0"
I20260927 14:57:10.203989 5607 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 14:57:10.206980 5621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:10.208369 5607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 14:57:10.208513 5607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "413f79d942cf4524906ff800d6b621ea"
format_stamp: "Formatted at 2026-09-27 14:57:10 on dist-test-slave-pz0h"
server_key: "8cfb7aeadcda20b9b9f1156d51b30bff"
server_key_iv: "6aea589b07343b0dfd4fcdadd5aa4ee7"
server_key_version: "encryptionkey@0"
I20260927 14:57:10.208639 5607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:10.229666 5607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:10.230437 5607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:10.230648 5607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:10.237448 5607 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35229
I20260927 14:57:10.237468 5673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35229 every 8 connection(s)
I20260927 14:57:10.238440 5607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:57:10.239914 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 5607
I20260927 14:57:10.240041 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 14:57:10.240355 392 external_mini_cluster.cc:1428] Setting key a6d150c0f6f00a9393db3f477b9921d5
I20260927 14:57:10.242245 5674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:10.247960 5674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 413f79d942cf4524906ff800d6b621ea: Bootstrap starting.
I20260927 14:57:10.249599 5677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:10.241828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41130 (local address 127.0.98.62:35229)
0927 14:57:10.242586 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:10.242603 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:10.242642 (+ 39us) server_negotiation.cc:407] Connection header received
0927 14:57:10.243356 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:10.243376 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:10.243794 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:10.244142 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:10.244566 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:10.245543 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:10.246402 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:10.246914 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:10.247064 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:10.247871 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:10.247876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:10.247890 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:57:10.248317 (+ 427us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:10.248376 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:57:10.248396 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:10.248481 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:10.248627 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 14:57:10.248884 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":145,"threads_started":1}
I20260927 14:57:10.250264 5674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 413f79d942cf4524906ff800d6b621ea: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:10.251163 5674 log.cc:824] T 00000000000000000000000000000000 P 413f79d942cf4524906ff800d6b621ea: Log is configured to *not* fsync() on all Append() calls
I20260927 14:57:10.252550 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:57:10.253571 5674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 413f79d942cf4524906ff800d6b621ea: No bootstrap required, opened a new log
I20260927 14:57:10.257490 5674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 413f79d942cf4524906ff800d6b621ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "413f79d942cf4524906ff800d6b621ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35229 } }
I20260927 14:57:10.257771 5674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 413f79d942cf4524906ff800d6b621ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:10.257808 5674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 413f79d942cf4524906ff800d6b621ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 413f79d942cf4524906ff800d6b621ea, State: Initialized, Role: FOLLOWER
I20260927 14:57:10.258247 5674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 413f79d942cf4524906ff800d6b621ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "413f79d942cf4524906ff800d6b621ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35229 } }
I20260927 14:57:10.258384 5674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 413f79d942cf4524906ff800d6b621ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:57:10.258435 5674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 413f79d942cf4524906ff800d6b621ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:57:10.258515 5674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 413f79d942cf4524906ff800d6b621ea [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:10.259586 5674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 413f79d942cf4524906ff800d6b621ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "413f79d942cf4524906ff800d6b621ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35229 } }
I20260927 14:57:10.259945 5674 leader_election.cc:302] T 00000000000000000000000000000000 P 413f79d942cf4524906ff800d6b621ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 413f79d942cf4524906ff800d6b621ea; no voters:
I20260927 14:57:10.260267 5674 leader_election.cc:288] T 00000000000000000000000000000000 P 413f79d942cf4524906ff800d6b621ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:57:10.260399 5679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 413f79d942cf4524906ff800d6b621ea [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:57:10.260625 5679 raft_consensus.cc:686] T 00000000000000000000000000000000 P 413f79d942cf4524906ff800d6b621ea [term 1 LEADER]: Becoming Leader. State: Replica: 413f79d942cf4524906ff800d6b621ea, State: Running, Role: LEADER
I20260927 14:57:10.260994 5679 consensus_queue.cc:237] T 00000000000000000000000000000000 P 413f79d942cf4524906ff800d6b621ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "413f79d942cf4524906ff800d6b621ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35229 } }
I20260927 14:57:10.261546 5674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 413f79d942cf4524906ff800d6b621ea [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:57:10.262825 5681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 413f79d942cf4524906ff800d6b621ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 413f79d942cf4524906ff800d6b621ea. Latest consensus state: current_term: 1 leader_uuid: "413f79d942cf4524906ff800d6b621ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "413f79d942cf4524906ff800d6b621ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35229 } } }
I20260927 14:57:10.262768 5680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 413f79d942cf4524906ff800d6b621ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "413f79d942cf4524906ff800d6b621ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "413f79d942cf4524906ff800d6b621ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35229 } } }
I20260927 14:57:10.262953 5681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 413f79d942cf4524906ff800d6b621ea [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:10.262954 5680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 413f79d942cf4524906ff800d6b621ea [sys.catalog]: This master's current role is: LEADER
W20260927 14:57:10.266147 5694 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 413f79d942cf4524906ff800d6b621ea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:57:10.266233 5694 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:57:10.266317 5689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:57:10.267822 5689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:57:10.273993 5689 catalog_manager.cc:1347] Generated new cluster ID: 064ed7a99e1c4477941d56bf7141fc3f
I20260927 14:57:10.274082 5689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:57:10.292408 5689 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:57:10.293761 5689 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:57:10.307717 5689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 413f79d942cf4524906ff800d6b621ea: Generated new TSK 0
I20260927 14:57:10.308739 5689 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 14:57:10.348474 5704 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 14:57:10.348574 5704 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 14:57:10.349488 5677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:10.325933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41136 (local address 127.0.98.62:35229)
0927 14:57:10.326067 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:10.326072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:10.326325 (+ 253us) server_negotiation.cc:407] Connection header received
0927 14:57:10.326495 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:10.326497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:10.326554 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:10.326644 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0927 14:57:10.326922 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:10.327423 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:10.328668 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:10.328909 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:10.328995 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:10.329308 (+ 313us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 14:57:10.349057 (+ 19749us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 14:57:10.349344 (+ 287us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41136: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 14:57:10.349555 5677 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41136: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 14:57:10.349679 5703 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:10.325792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35229 (local address 127.0.0.1:41136)
0927 14:57:10.326221 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:10.326236 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:10.326369 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:10.326663 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:10.326665 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:10.326673 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 14:57:10.326823 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:10.326830 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:10.327536 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:10.327539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:10.328527 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:10.328539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:10.329004 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:10.329027 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:10.329119 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:10.329149 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 14:57:10.349304 (+ 20155us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 14:57:10.349500 (+ 196us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35229: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":116,"threads_started":1}
I20260927 14:57:10.350510 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 5607
2026-09-27T14:57:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (347 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 14:57:10.389197 392 mini_oidc.cc:152] Starting JWKS server
I20260927 14:57:10.390059 392 webserver.cc:460] Webserver started at https://127.0.0.1:38357/ using document root <none> and password file <none>
I20260927 14:57:10.390148 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 14:57:10.390368 392 webserver.cc:460] Webserver started at http://127.0.0.1:43697/ using document root <none> and password file <none>
2026-09-27T14:57:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:57:10Z Disabled control of system clock
I20260927 14:57:10.398684 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42253
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42253
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38357/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 14:57:10.511276 5710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:10.511580 5710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:10.511622 5710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:10.515821 5710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:57:10.515904 5710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:10.515919 5710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:10.515935 5710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:57:10.515950 5710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 14:57:10.515966 5710 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 14:57:10.515990 5710 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38357/jwks.json
I20260927 14:57:10.521046 5710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42253
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38357/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:10.522151 5710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:10.523702 5710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:10.532097 5718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:10.532100 5716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:10.532100 5715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:10.532321 5710 server_base.cc:1034] running on GCE node
I20260927 14:57:10.533005 5710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:10.534159 5710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:10.535349 5710 hybrid_clock.cc:648] HybridClock initialized: now 1790521030535327 us; error 42 us; skew 500 ppm
I20260927 14:57:10.537380 5710 webserver.cc:460] Webserver started at http://127.0.98.62:34955/ using document root <none> and password file <none>
I20260927 14:57:10.537966 5710 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:10.538043 5710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:10.538215 5710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:10.540403 5710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "69f1edb3a33849ffa6a8210b908a141c"
format_stamp: "Formatted at 2026-09-27 14:57:10 on dist-test-slave-pz0h"
server_key: "93207fd21ebc8362bb6fe039cb248038"
server_key_iv: "0145e4ff458197bfa309f7686e351ace"
server_key_version: "encryptionkey@0"
I20260927 14:57:10.540963 5710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "69f1edb3a33849ffa6a8210b908a141c"
format_stamp: "Formatted at 2026-09-27 14:57:10 on dist-test-slave-pz0h"
server_key: "93207fd21ebc8362bb6fe039cb248038"
server_key_iv: "0145e4ff458197bfa309f7686e351ace"
server_key_version: "encryptionkey@0"
I20260927 14:57:10.545298 5710 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 14:57:10.548343 5724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:10.549860 5710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 14:57:10.550007 5710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "69f1edb3a33849ffa6a8210b908a141c"
format_stamp: "Formatted at 2026-09-27 14:57:10 on dist-test-slave-pz0h"
server_key: "93207fd21ebc8362bb6fe039cb248038"
server_key_iv: "0145e4ff458197bfa309f7686e351ace"
server_key_version: "encryptionkey@0"
I20260927 14:57:10.550127 5710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:10.572256 5710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:10.572978 5710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:10.573139 5710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:10.579994 5710 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42253
I20260927 14:57:10.580011 5776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42253 every 8 connection(s)
I20260927 14:57:10.580982 5710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:57:10.584542 5777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:10.584990 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 5710
I20260927 14:57:10.585093 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 14:57:10.585345 392 external_mini_cluster.cc:1428] Setting key b90a55f83496a9489145ca13e10eaa12
I20260927 14:57:10.590773 5777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69f1edb3a33849ffa6a8210b908a141c: Bootstrap starting.
I20260927 14:57:10.593200 5777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69f1edb3a33849ffa6a8210b908a141c: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:10.593982 5777 log.cc:824] T 00000000000000000000000000000000 P 69f1edb3a33849ffa6a8210b908a141c: Log is configured to *not* fsync() on all Append() calls
I20260927 14:57:10.595026 5780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:10.586842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52326 (local address 127.0.98.62:42253)
0927 14:57:10.587563 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:10.587580 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:10.587619 (+ 39us) server_negotiation.cc:407] Connection header received
0927 14:57:10.588397 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:10.588424 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:10.588876 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:10.589221 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:10.589676 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:10.590726 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:10.591674 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:10.592159 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:10.592320 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:10.593126 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:10.593131 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:10.593146 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:57:10.593603 (+ 457us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:10.593665 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:57:10.593688 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:10.593775 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:10.593921 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 14:57:10.594190 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":128,"threads_started":1}
I20260927 14:57:10.596272 5777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69f1edb3a33849ffa6a8210b908a141c: No bootstrap required, opened a new log
I20260927 14:57:10.597971 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:57:10.600132 5777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69f1edb3a33849ffa6a8210b908a141c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f1edb3a33849ffa6a8210b908a141c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42253 } }
I20260927 14:57:10.600395 5777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69f1edb3a33849ffa6a8210b908a141c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:10.600461 5777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69f1edb3a33849ffa6a8210b908a141c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f1edb3a33849ffa6a8210b908a141c, State: Initialized, Role: FOLLOWER
I20260927 14:57:10.600883 5777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69f1edb3a33849ffa6a8210b908a141c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f1edb3a33849ffa6a8210b908a141c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42253 } }
I20260927 14:57:10.601018 5777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69f1edb3a33849ffa6a8210b908a141c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:57:10.601084 5777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69f1edb3a33849ffa6a8210b908a141c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:57:10.601184 5777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69f1edb3a33849ffa6a8210b908a141c [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:10.602169 5777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69f1edb3a33849ffa6a8210b908a141c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f1edb3a33849ffa6a8210b908a141c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42253 } }
I20260927 14:57:10.602555 5777 leader_election.cc:302] T 00000000000000000000000000000000 P 69f1edb3a33849ffa6a8210b908a141c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69f1edb3a33849ffa6a8210b908a141c; no voters:
I20260927 14:57:10.602948 5777 leader_election.cc:288] T 00000000000000000000000000000000 P 69f1edb3a33849ffa6a8210b908a141c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:57:10.603129 5782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69f1edb3a33849ffa6a8210b908a141c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:57:10.603348 5782 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69f1edb3a33849ffa6a8210b908a141c [term 1 LEADER]: Becoming Leader. State: Replica: 69f1edb3a33849ffa6a8210b908a141c, State: Running, Role: LEADER
I20260927 14:57:10.603674 5782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69f1edb3a33849ffa6a8210b908a141c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f1edb3a33849ffa6a8210b908a141c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42253 } }
I20260927 14:57:10.604257 5777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69f1edb3a33849ffa6a8210b908a141c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:57:10.605226 5784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69f1edb3a33849ffa6a8210b908a141c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69f1edb3a33849ffa6a8210b908a141c. Latest consensus state: current_term: 1 leader_uuid: "69f1edb3a33849ffa6a8210b908a141c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f1edb3a33849ffa6a8210b908a141c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42253 } } }
I20260927 14:57:10.605345 5784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69f1edb3a33849ffa6a8210b908a141c [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:10.605669 5783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69f1edb3a33849ffa6a8210b908a141c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69f1edb3a33849ffa6a8210b908a141c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f1edb3a33849ffa6a8210b908a141c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42253 } } }
I20260927 14:57:10.605731 5791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:57:10.605773 5783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69f1edb3a33849ffa6a8210b908a141c [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:10.609144 5791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:57:10.615206 5791 catalog_manager.cc:1347] Generated new cluster ID: 6d35d04f9c44460e83f0dff417f62a16
I20260927 14:57:10.615309 5791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:57:10.639235 5791 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:57:10.640345 5791 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:57:10.647789 5791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69f1edb3a33849ffa6a8210b908a141c: Generated new TSK 0
I20260927 14:57:10.648548 5791 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:57:10.664693 5780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:10.660187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52330 (local address 127.0.98.62:42253)
0927 14:57:10.660308 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:10.660313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:10.660473 (+ 160us) server_negotiation.cc:407] Connection header received
0927 14:57:10.660648 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:10.660651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:10.660701 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:10.660797 (+ 96us) server_negotiation.cc:226] Negotiated authn=JWT
0927 14:57:10.661023 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:10.661574 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:10.662455 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:10.662685 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:10.662773 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:10.663103 (+ 330us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 14:57:10.664314 (+ 1211us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38357/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 14:57:10.664546 (+ 232us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52330: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38357/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 14:57:10.664778 5780 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52330: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38357/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 14:57:10.665059 5806 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:10.660104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42253 (local address 127.0.0.1:52330)
0927 14:57:10.660390 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:10.660402 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:10.660513 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:10.660804 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:10.660806 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:10.660813 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0927 14:57:10.660924 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:10.660930 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:10.661683 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:10.661686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:10.662313 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:10.662346 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:10.662773 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:10.662825 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:10.662915 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:10.662940 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 14:57:10.664467 (+ 1527us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38357/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 14:57:10.664611 (+ 144us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42253: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38357/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":114,"threads_started":1}
I20260927 14:57:10.665656 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 5710
E20260927 14:57:10.675801 5807 webserver.cc:533] Webserver: sslize failed:
E20260927 14:57:10.675908 5807 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T14:57:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (314 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T14:57:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:57:10Z Disabled control of system clock
I20260927 14:57:10.712890 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42961
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42961
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521004237705-392-0/testchainca.cert with env {}
W20260927 14:57:10.827574 5811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:10.827881 5811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:10.827940 5811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:10.832186 5811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:57:10.832265 5811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:10.832280 5811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:10.832296 5811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:57:10.832311 5811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 14:57:10.832326 5811 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 14:57:10.832338 5811 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 14:57:10.838034 5811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42961
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521004237705-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:10.839226 5811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:10.840226 5811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:10.849042 5817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:10.849021 5816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:10.849020 5819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:10.849063 5811 server_base.cc:1034] running on GCE node
I20260927 14:57:10.849825 5811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:10.851176 5811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:10.852362 5811 hybrid_clock.cc:648] HybridClock initialized: now 1790521030852341 us; error 38 us; skew 500 ppm
I20260927 14:57:10.854404 5811 webserver.cc:460] Webserver started at http://127.0.98.62:33157/ using document root <none> and password file <none>
I20260927 14:57:10.855062 5811 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:10.855141 5811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:10.855357 5811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:10.857522 5811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "87c2c009ce974680b4f4bf84e472ad9c"
format_stamp: "Formatted at 2026-09-27 14:57:10 on dist-test-slave-pz0h"
server_key: "eb1da77c0801ba8d986d03543d43eea8"
server_key_iv: "ff3686e11631a634a17b9493e9ae847e"
server_key_version: "encryptionkey@0"
I20260927 14:57:10.858107 5811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "87c2c009ce974680b4f4bf84e472ad9c"
format_stamp: "Formatted at 2026-09-27 14:57:10 on dist-test-slave-pz0h"
server_key: "eb1da77c0801ba8d986d03543d43eea8"
server_key_iv: "ff3686e11631a634a17b9493e9ae847e"
server_key_version: "encryptionkey@0"
I20260927 14:57:10.862056 5811 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 14:57:10.864915 5825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:10.866403 5811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 14:57:10.866572 5811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "87c2c009ce974680b4f4bf84e472ad9c"
format_stamp: "Formatted at 2026-09-27 14:57:10 on dist-test-slave-pz0h"
server_key: "eb1da77c0801ba8d986d03543d43eea8"
server_key_iv: "ff3686e11631a634a17b9493e9ae847e"
server_key_version: "encryptionkey@0"
I20260927 14:57:10.866703 5811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:10.876878 5811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:10.877616 5811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:10.877797 5811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:10.884625 5811 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42961
I20260927 14:57:10.884639 5877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42961 every 8 connection(s)
I20260927 14:57:10.885635 5811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:57:10.889037 5878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:10.889292 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 5811
I20260927 14:57:10.889408 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 14:57:10.889706 392 external_mini_cluster.cc:1428] Setting key c1378d56222b90a7b247297e1769c482
I20260927 14:57:10.896116 5878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c2c009ce974680b4f4bf84e472ad9c: Bootstrap starting.
I20260927 14:57:10.898489 5878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87c2c009ce974680b4f4bf84e472ad9c: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:10.899209 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:10.891197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48858 (local address 127.0.98.62:42961)
0927 14:57:10.891915 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:10.891932 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:10.891973 (+ 41us) server_negotiation.cc:407] Connection header received
0927 14:57:10.892690 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:10.892711 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:10.893162 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:10.893514 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:10.893941 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:10.895004 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:10.895877 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:10.896395 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:10.896553 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:10.897380 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:10.897386 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:10.897400 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:57:10.897804 (+ 404us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:10.897878 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:57:10.897900 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:10.897988 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:10.898135 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 14:57:10.898401 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":155,"threads_started":1}
I20260927 14:57:10.899278 5878 log.cc:824] T 00000000000000000000000000000000 P 87c2c009ce974680b4f4bf84e472ad9c: Log is configured to *not* fsync() on all Append() calls
I20260927 14:57:10.901595 5878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c2c009ce974680b4f4bf84e472ad9c: No bootstrap required, opened a new log
I20260927 14:57:10.902252 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:57:10.905898 5878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87c2c009ce974680b4f4bf84e472ad9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c2c009ce974680b4f4bf84e472ad9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } }
I20260927 14:57:10.906174 5878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87c2c009ce974680b4f4bf84e472ad9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:10.906224 5878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87c2c009ce974680b4f4bf84e472ad9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c2c009ce974680b4f4bf84e472ad9c, State: Initialized, Role: FOLLOWER
I20260927 14:57:10.906672 5878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87c2c009ce974680b4f4bf84e472ad9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c2c009ce974680b4f4bf84e472ad9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } }
I20260927 14:57:10.906850 5878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87c2c009ce974680b4f4bf84e472ad9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:57:10.906921 5878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87c2c009ce974680b4f4bf84e472ad9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:57:10.907023 5878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87c2c009ce974680b4f4bf84e472ad9c [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:10.907996 5878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87c2c009ce974680b4f4bf84e472ad9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c2c009ce974680b4f4bf84e472ad9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } }
I20260927 14:57:10.908349 5878 leader_election.cc:302] T 00000000000000000000000000000000 P 87c2c009ce974680b4f4bf84e472ad9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87c2c009ce974680b4f4bf84e472ad9c; no voters:
I20260927 14:57:10.908715 5878 leader_election.cc:288] T 00000000000000000000000000000000 P 87c2c009ce974680b4f4bf84e472ad9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:57:10.908838 5883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87c2c009ce974680b4f4bf84e472ad9c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:57:10.909051 5883 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87c2c009ce974680b4f4bf84e472ad9c [term 1 LEADER]: Becoming Leader. State: Replica: 87c2c009ce974680b4f4bf84e472ad9c, State: Running, Role: LEADER
I20260927 14:57:10.909369 5883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87c2c009ce974680b4f4bf84e472ad9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c2c009ce974680b4f4bf84e472ad9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } }
I20260927 14:57:10.910012 5878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87c2c009ce974680b4f4bf84e472ad9c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:57:10.911120 5885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c2c009ce974680b4f4bf84e472ad9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87c2c009ce974680b4f4bf84e472ad9c. Latest consensus state: current_term: 1 leader_uuid: "87c2c009ce974680b4f4bf84e472ad9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c2c009ce974680b4f4bf84e472ad9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } } }
I20260927 14:57:10.911124 5884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c2c009ce974680b4f4bf84e472ad9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87c2c009ce974680b4f4bf84e472ad9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c2c009ce974680b4f4bf84e472ad9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } } }
I20260927 14:57:10.911278 5885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c2c009ce974680b4f4bf84e472ad9c [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:10.911278 5884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c2c009ce974680b4f4bf84e472ad9c [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:10.911716 5892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:57:10.915100 5892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:57:10.922130 5892 catalog_manager.cc:1347] Generated new cluster ID: b947f8890427441a8b32332724232528
I20260927 14:57:10.922219 5892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:57:10.938714 5892 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:57:10.939729 5892 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:57:10.949836 5892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87c2c009ce974680b4f4bf84e472ad9c: Generated new TSK 0
I20260927 14:57:10.950568 5892 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:57:11.451004 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:10.967347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48870 (local address 127.0.98.62:42961)
0927 14:57:10.967472 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:10.967476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:10.967667 (+ 191us) server_negotiation.cc:407] Connection header received
0927 14:57:10.967857 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:10.967860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:10.967919 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:10.968013 (+ 94us) server_negotiation.cc:226] Negotiated authn=JWT
0927 14:57:10.968247 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:10.968785 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:10.969669 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:10.969892 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:10.969985 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:10.970307 (+ 322us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 14:57:11.450496 (+480189us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 14:57:11.450839 (+ 343us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48870: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 14:57:11.451094 5881 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48870: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 14:57:11.451122 5907 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:10.967267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42961 (local address 127.0.0.1:48870)
0927 14:57:10.967578 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:10.967590 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:10.967709 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:10.968027 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:10.968031 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:10.968039 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 14:57:10.968150 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:10.968156 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:10.968898 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:10.968901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:10.969534 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:10.969543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:10.969996 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:10.970022 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:10.970113 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:10.970141 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 14:57:11.450745 (+480604us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 14:57:11.450957 (+ 212us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42961: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":116,"threads_started":1}
I20260927 14:57:11.451550 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 5811
2026-09-27T14:57:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (765 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (125 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (124 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:57:11 dist-test-slave-pz0h krb5kdc[5919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:57:11 dist-test-slave-pz0h krb5kdc[5919](info): set up 2 sockets
Sep 27 14:57:11 dist-test-slave-pz0h krb5kdc[5919](info): commencing operation
krb5kdc: starting...
W20260927 14:57:12.658833 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 14:57:12 dist-test-slave-pz0h krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:57:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:57:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:57:12.773375 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46547
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:12.888175 5932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:12.888463 5932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:12.888520 5932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:12.892724 5932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:57:12.892799 5932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:12.892827 5932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:12.892849 5932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:57:12.892876 5932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:57:12.898567 5932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:12.899796 5932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:12.900856 5932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 14:57:12.909544 5932 server_base.cc:1034] running on GCE node
W20260927 14:57:12.909485 5937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:12.909502 5940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:12.909507 5938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:12.910355 5932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:12.911571 5932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:12.912766 5932 hybrid_clock.cc:648] HybridClock initialized: now 1790521032912751 us; error 32 us; skew 500 ppm
Sep 27 14:57:12 dist-test-slave-pz0h krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:12.921350 5932 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:57:12.922326 5932 webserver.cc:460] Webserver started at http://127.0.98.62:41775/ using document root <none> and password file <none>
I20260927 14:57:12.922953 5932 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:12.923028 5932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:12.923210 5932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:12.925323 5932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "8952da33417b43c9bc99d85354d7d6e5"
format_stamp: "Formatted at 2026-09-27 14:57:12 on dist-test-slave-pz0h"
server_key: "755c4bde4058d33e8a21fb7494dc0fa5"
server_key_iv: "46026e41c412f0f717f7ccf1bee8221c"
server_key_version: "encryptionkey@0"
I20260927 14:57:12.925876 5932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "8952da33417b43c9bc99d85354d7d6e5"
format_stamp: "Formatted at 2026-09-27 14:57:12 on dist-test-slave-pz0h"
server_key: "755c4bde4058d33e8a21fb7494dc0fa5"
server_key_iv: "46026e41c412f0f717f7ccf1bee8221c"
server_key_version: "encryptionkey@0"
I20260927 14:57:12.929975 5932 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 14:57:12.932761 5947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:12.934083 5932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:57:12.934202 5932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "8952da33417b43c9bc99d85354d7d6e5"
format_stamp: "Formatted at 2026-09-27 14:57:12 on dist-test-slave-pz0h"
server_key: "755c4bde4058d33e8a21fb7494dc0fa5"
server_key_iv: "46026e41c412f0f717f7ccf1bee8221c"
server_key_version: "encryptionkey@0"
I20260927 14:57:12.934306 5932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:12.956410 5932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:12.959681 5932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:12.959849 5932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:12.966921 5932 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46547
I20260927 14:57:12.967175 5999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46547 every 8 connection(s)
I20260927 14:57:12.967917 5932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:57:12.970278 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 5932
I20260927 14:57:12.970381 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 14:57:12.970638 392 external_mini_cluster.cc:1428] Setting key 5f7661f46a72f914a00bd15ebef6258f
I20260927 14:57:12.970885 6000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:12.977401 6000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8952da33417b43c9bc99d85354d7d6e5: Bootstrap starting.
Sep 27 14:57:12 dist-test-slave-pz0h krb5kdc[5919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:57:12.980329 6000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8952da33417b43c9bc99d85354d7d6e5: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:12.981081 6000 log.cc:824] T 00000000000000000000000000000000 P 8952da33417b43c9bc99d85354d7d6e5: Log is configured to *not* fsync() on all Append() calls
I20260927 14:57:12.983142 6000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8952da33417b43c9bc99d85354d7d6e5: No bootstrap required, opened a new log
I20260927 14:57:12.984444 6003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:12.971994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56398 (local address 127.0.98.62:46547)
0927 14:57:12.972450 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:12.972466 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:12.972503 (+ 37us) server_negotiation.cc:407] Connection header received
0927 14:57:12.973167 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:12.973185 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:12.973524 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:12.973846 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:12.974643 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:12.975699 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:12.976528 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:12.977013 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:12.977167 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:12.979885 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:12.979891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:12.979906 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:12.979940 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:12.981625 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:12.982078 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:12.982083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:12.982089 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:12.982199 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:12.982467 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:12.982470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:12.982472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:12.982925 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:12.983114 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:12.983388 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 14:57:12.983684 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":157,"threads_started":1}
I20260927 14:57:12.987308 6000 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8952da33417b43c9bc99d85354d7d6e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8952da33417b43c9bc99d85354d7d6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46547 } }
I20260927 14:57:12.987571 6000 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8952da33417b43c9bc99d85354d7d6e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:12.987627 6000 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8952da33417b43c9bc99d85354d7d6e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8952da33417b43c9bc99d85354d7d6e5, State: Initialized, Role: FOLLOWER
I20260927 14:57:12.988047 6000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8952da33417b43c9bc99d85354d7d6e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8952da33417b43c9bc99d85354d7d6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46547 } }
I20260927 14:57:12.988180 6000 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8952da33417b43c9bc99d85354d7d6e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:57:12.988246 6000 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8952da33417b43c9bc99d85354d7d6e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:57:12.988344 6000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8952da33417b43c9bc99d85354d7d6e5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:12.989349 6000 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8952da33417b43c9bc99d85354d7d6e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8952da33417b43c9bc99d85354d7d6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46547 } }
I20260927 14:57:12.989697 6000 leader_election.cc:302] T 00000000000000000000000000000000 P 8952da33417b43c9bc99d85354d7d6e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8952da33417b43c9bc99d85354d7d6e5; no voters:
I20260927 14:57:12.990036 6000 leader_election.cc:288] T 00000000000000000000000000000000 P 8952da33417b43c9bc99d85354d7d6e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:57:12.990182 6005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8952da33417b43c9bc99d85354d7d6e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:57:12.990422 6005 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8952da33417b43c9bc99d85354d7d6e5 [term 1 LEADER]: Becoming Leader. State: Replica: 8952da33417b43c9bc99d85354d7d6e5, State: Running, Role: LEADER
I20260927 14:57:12.990772 6005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8952da33417b43c9bc99d85354d7d6e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8952da33417b43c9bc99d85354d7d6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46547 } }
I20260927 14:57:12.991420 6000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8952da33417b43c9bc99d85354d7d6e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:57:12.992561 6007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8952da33417b43c9bc99d85354d7d6e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8952da33417b43c9bc99d85354d7d6e5. Latest consensus state: current_term: 1 leader_uuid: "8952da33417b43c9bc99d85354d7d6e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8952da33417b43c9bc99d85354d7d6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46547 } } }
I20260927 14:57:12.992576 6006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8952da33417b43c9bc99d85354d7d6e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8952da33417b43c9bc99d85354d7d6e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8952da33417b43c9bc99d85354d7d6e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46547 } } }
I20260927 14:57:12.992728 6007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8952da33417b43c9bc99d85354d7d6e5 [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:12.992730 6006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8952da33417b43c9bc99d85354d7d6e5 [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:12.993259 6014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:57:12.996680 6014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:57:13.003062 6014 catalog_manager.cc:1347] Generated new cluster ID: efc7ceaa8150442788fa8a75286f1e1c
I20260927 14:57:13.003142 6014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:57:13.021044 6014 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:57:13.022207 6014 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:57:13.031824 6014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8952da33417b43c9bc99d85354d7d6e5: Generated new TSK 0
I20260927 14:57:13.032747 6014 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:57:13.061668 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46547
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:13.175247 6026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:13.175531 6026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:13.175575 6026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:13.180027 6026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:13.180114 6026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:13.180179 6026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:57:13.186112 6026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:13.187309 6026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:13.188337 6026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:13.198567 6032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:13.198887 6026 server_base.cc:1034] running on GCE node
W20260927 14:57:13.198621 6031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:13.198639 6034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:13.199571 6026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:13.200265 6026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:13.201447 6026 hybrid_clock.cc:648] HybridClock initialized: now 1790521033201430 us; error 36 us; skew 500 ppm
Sep 27 14:57:13 dist-test-slave-pz0h krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:13.204883 6026 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:57:13.205897 6026 webserver.cc:460] Webserver started at http://127.0.98.1:41125/ using document root <none> and password file <none>
I20260927 14:57:13.206459 6026 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:13.206522 6026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:13.206697 6026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:13.208885 6026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-0/data/instance:
uuid: "127768b12c1d4f64a693c51b33595f94"
format_stamp: "Formatted at 2026-09-27 14:57:13 on dist-test-slave-pz0h"
server_key: "1fc56816a983ffaa94546bf55747e4ac"
server_key_iv: "f27c1a3fe7e266779fd603cff19e6125"
server_key_version: "encryptionkey@0"
I20260927 14:57:13.209453 6026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "127768b12c1d4f64a693c51b33595f94"
format_stamp: "Formatted at 2026-09-27 14:57:13 on dist-test-slave-pz0h"
server_key: "1fc56816a983ffaa94546bf55747e4ac"
server_key_iv: "f27c1a3fe7e266779fd603cff19e6125"
server_key_version: "encryptionkey@0"
I20260927 14:57:13.213613 6026 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 14:57:13.216522 6041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:13.217877 6026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 14:57:13.218003 6026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "127768b12c1d4f64a693c51b33595f94"
format_stamp: "Formatted at 2026-09-27 14:57:13 on dist-test-slave-pz0h"
server_key: "1fc56816a983ffaa94546bf55747e4ac"
server_key_iv: "f27c1a3fe7e266779fd603cff19e6125"
server_key_version: "encryptionkey@0"
I20260927 14:57:13.218109 6026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:13.244208 6026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:13.248457 6026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:13.248637 6026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:13.249222 6026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:13.250196 6026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:13.250258 6026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:13.250301 6026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:13.250315 6026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:13.259325 6026 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41401
I20260927 14:57:13.259354 6154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41401 every 8 connection(s)
I20260927 14:57:13.260315 6026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:57:13.268618 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 6026
I20260927 14:57:13.268754 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:57:13.269045 392 external_mini_cluster.cc:1428] Setting key 35ef423c83a9d580be7e41df7d6dce86
Sep 27 14:57:13 dist-test-slave-pz0h krb5kdc[5919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:57:13.276911 6003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:13.262339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37565 (local address 127.0.98.62:46547)
0927 14:57:13.262534 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:13.262539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:13.263487 (+ 948us) server_negotiation.cc:407] Connection header received
0927 14:57:13.264944 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:13.264947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:13.265004 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:13.265091 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:13.266669 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:13.267294 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:13.268606 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:13.268957 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:13.269075 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:13.272319 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:13.272324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:13.272327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:13.272365 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:13.273962 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:13.274586 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:13.274590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:13.274592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:13.274689 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:13.275155 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:13.275159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:13.275161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:13.275636 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:13.275945 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:13.276539 (+ 594us) server_negotiation.cc:299] Negotiation successful
0927 14:57:13.276710 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 14:57:13.277755 6157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:13.262771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46547 (local address 127.0.98.1:37565)
0927 14:57:13.263306 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:13.263352 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:13.264657 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:13.265291 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:13.265321 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:13.265759 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:13.266435 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:13.266457 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:13.267456 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:13.267459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:13.268419 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:13.268431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:13.269162 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:13.269167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:13.269339 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:13.270016 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:13.270040 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:13.272105 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:13.274126 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:13.274132 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:13.274148 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:13.274478 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:13.274853 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:13.274857 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:13.274864 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:13.275039 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:13.275921 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:13.275928 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:13.276230 (+ 302us) client_negotiation.cc:769] Sending connection context
0927 14:57:13.276426 (+ 196us) client_negotiation.cc:240] Negotiation successful
0927 14:57:13.276738 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":136,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 14:57:13.279151 6155 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46547
I20260927 14:57:13.279431 6155 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:13.280184 6155 heartbeater.cc:511] Master 127.0.98.62:46547 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 14:57:13.282153 5964 ts_manager.cc:194] Registered new tserver with Master: 127768b12c1d4f64a693c51b33595f94 (127.0.98.1:41401)
I20260927 14:57:13.283672 5964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:57:13.298498 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46547
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:13.420037 6160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:13.420341 6160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:13.420403 6160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:13.425951 6160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:13.426059 6160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:13.426154 6160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:57:13.432458 6160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:13.433777 6160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:13.434942 6160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:13.445318 6165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:13.445318 6168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:13.445320 6166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:13.445513 6160 server_base.cc:1034] running on GCE node
I20260927 14:57:13.446053 6160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:13.446759 6160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:13.447980 6160 hybrid_clock.cc:648] HybridClock initialized: now 1790521033447946 us; error 50 us; skew 500 ppm
Sep 27 14:57:13 dist-test-slave-pz0h krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:13.451231 6160 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:57:13.452589 6160 webserver.cc:460] Webserver started at http://127.0.98.2:37037/ using document root <none> and password file <none>
I20260927 14:57:13.453226 6160 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:13.453310 6160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:13.453538 6160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:13.455900 6160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-1/data/instance:
uuid: "4145b1ea578b4da9bcebb8eda7ae4935"
format_stamp: "Formatted at 2026-09-27 14:57:13 on dist-test-slave-pz0h"
server_key: "e40b5f5d5f7035c00f414d0597b6a512"
server_key_iv: "ee8eea471c49b1dd13291e6349e887ae"
server_key_version: "encryptionkey@0"
I20260927 14:57:13.456508 6160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4145b1ea578b4da9bcebb8eda7ae4935"
format_stamp: "Formatted at 2026-09-27 14:57:13 on dist-test-slave-pz0h"
server_key: "e40b5f5d5f7035c00f414d0597b6a512"
server_key_iv: "ee8eea471c49b1dd13291e6349e887ae"
server_key_version: "encryptionkey@0"
I20260927 14:57:13.462335 6160 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 14:57:13.465233 6175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:13.466596 6160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:57:13.466740 6160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "4145b1ea578b4da9bcebb8eda7ae4935"
format_stamp: "Formatted at 2026-09-27 14:57:13 on dist-test-slave-pz0h"
server_key: "e40b5f5d5f7035c00f414d0597b6a512"
server_key_iv: "ee8eea471c49b1dd13291e6349e887ae"
server_key_version: "encryptionkey@0"
I20260927 14:57:13.466917 6160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:13.499114 6160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:13.502243 6160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:13.502451 6160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:13.503122 6160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:13.504011 6160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:13.504061 6160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:13.504115 6160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:13.504141 6160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:13.513207 6160 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45699
I20260927 14:57:13.513278 6288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45699 every 8 connection(s)
I20260927 14:57:13.514287 6160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:57:13.515985 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 6160
I20260927 14:57:13.516084 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:57:13.516607 392 external_mini_cluster.cc:1428] Setting key ce217577755a1fea256b672fbd9c8f38
Sep 27 14:57:13 dist-test-slave-pz0h krb5kdc[5919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 14:57:13.531811 6003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:13.516244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38195 (local address 127.0.98.62:46547)
0927 14:57:13.516542 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:13.516548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:13.517674 (+ 1126us) server_negotiation.cc:407] Connection header received
0927 14:57:13.519970 (+ 2296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:13.519974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:13.520049 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:13.520170 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:13.522201 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:13.522985 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:13.524878 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:13.525119 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:13.525215 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:13.528244 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:13.528248 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:13.528250 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:13.528277 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:13.529681 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:13.530219 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:13.530221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:13.530222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:13.530290 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:13.530600 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:13.530603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:13.530604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:13.530855 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:13.530947 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:13.531499 (+ 552us) server_negotiation.cc:299] Negotiation successful
0927 14:57:13.531639 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260927 14:57:13.532752 6291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:13.516672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46547 (local address 127.0.98.2:38195)
0927 14:57:13.517356 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:13.517452 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:13.519584 (+ 2132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:13.520466 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:13.520481 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:13.521235 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:13.521963 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:13.521987 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:13.523169 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:13.523174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:13.524674 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:13.524693 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:13.525311 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:13.525316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:13.525551 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:13.526216 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:13.526240 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:13.528018 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:13.529808 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:13.529814 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:13.529830 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:13.530131 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:13.530365 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:13.530368 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:13.530370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:13.530517 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:13.530962 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:13.530970 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:13.531254 (+ 284us) client_negotiation.cc:769] Sending connection context
0927 14:57:13.531440 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 14:57:13.531835 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":137,"threads_started":1}
I20260927 14:57:13.534138 6289 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46547
I20260927 14:57:13.534384 6289 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:13.534940 6289 heartbeater.cc:511] Master 127.0.98.62:46547 requested a full tablet report, sending...
I20260927 14:57:13.536099 5964 ts_manager.cc:194] Registered new tserver with Master: 4145b1ea578b4da9bcebb8eda7ae4935 (127.0.98.2:45699)
I20260927 14:57:13.536916 5964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:57:13.546202 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46547
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:13.660286 6294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:13.660565 6294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:13.660609 6294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:13.664764 6294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:13.664840 6294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:13.664901 6294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:57:13.670571 6294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:13.671862 6294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:13.672971 6294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:13.682947 6302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:13.682983 6300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:13.682948 6299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:13.683269 6294 server_base.cc:1034] running on GCE node
I20260927 14:57:13.684082 6294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:13.685006 6294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:13.686218 6294 hybrid_clock.cc:648] HybridClock initialized: now 1790521033686190 us; error 46 us; skew 500 ppm
Sep 27 14:57:13 dist-test-slave-pz0h krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:13.689626 6294 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:57:13.690644 6294 webserver.cc:460] Webserver started at http://127.0.98.3:45643/ using document root <none> and password file <none>
I20260927 14:57:13.691355 6294 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:13.691432 6294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:13.691656 6294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:13.694092 6294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-2/data/instance:
uuid: "6aa67fb534924f2fb84ca241158b4477"
format_stamp: "Formatted at 2026-09-27 14:57:13 on dist-test-slave-pz0h"
server_key: "abc716fe0e47dd2cf8fbaf11745e4bc7"
server_key_iv: "d67df5490164df34710ca628bf0da3d9"
server_key_version: "encryptionkey@0"
I20260927 14:57:13.694734 6294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6aa67fb534924f2fb84ca241158b4477"
format_stamp: "Formatted at 2026-09-27 14:57:13 on dist-test-slave-pz0h"
server_key: "abc716fe0e47dd2cf8fbaf11745e4bc7"
server_key_iv: "d67df5490164df34710ca628bf0da3d9"
server_key_version: "encryptionkey@0"
I20260927 14:57:13.698949 6294 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 14:57:13.701915 6309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:13.703850 6294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:57:13.704056 6294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "6aa67fb534924f2fb84ca241158b4477"
format_stamp: "Formatted at 2026-09-27 14:57:13 on dist-test-slave-pz0h"
server_key: "abc716fe0e47dd2cf8fbaf11745e4bc7"
server_key_iv: "d67df5490164df34710ca628bf0da3d9"
server_key_version: "encryptionkey@0"
I20260927 14:57:13.704227 6294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:13.739907 6294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:13.743358 6294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:13.743564 6294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:13.744163 6294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:13.745012 6294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:13.745060 6294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:13.745114 6294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:13.745139 6294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:13.754097 6294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34577
I20260927 14:57:13.754164 6422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34577 every 8 connection(s)
I20260927 14:57:13.755062 6294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:57:13.763687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 6294
I20260927 14:57:13.763787 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:57:13.764132 392 external_mini_cluster.cc:1428] Setting key 81ed3cd4246df706d2d1853b5e7461ed
Sep 27 14:57:13 dist-test-slave-pz0h krb5kdc[5919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:57:13.769703 6003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:13.757222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48517 (local address 127.0.98.62:46547)
0927 14:57:13.757391 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:13.757395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:13.758270 (+ 875us) server_negotiation.cc:407] Connection header received
0927 14:57:13.759798 (+ 1528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:13.759801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:13.759865 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:13.759951 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:13.761244 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:13.761788 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:13.763007 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:13.763233 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:13.763328 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:13.766055 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:13.766058 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:13.766060 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:13.766085 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:13.767551 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:13.768108 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:13.768112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:13.768113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:13.768189 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:13.768529 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:13.768533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:13.768535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:13.768730 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:13.768815 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:13.769383 (+ 568us) server_negotiation.cc:299] Negotiation successful
0927 14:57:13.769526 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 14:57:13.770695 6425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:13.757599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46547 (local address 127.0.98.3:48517)
0927 14:57:13.758109 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:13.758162 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:13.759539 (+ 1377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:13.760130 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:13.760138 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:13.760538 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:13.761072 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:13.761086 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:13.761892 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:13.761895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:13.762866 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:13.762878 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:13.763411 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:13.763416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:13.763573 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:13.764223 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:13.764244 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:13.765882 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:13.767692 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:13.767700 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:13.767721 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:13.768009 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:13.768271 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:13.768275 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:13.768277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:13.768442 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:13.768825 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:13.768832 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:13.769114 (+ 282us) client_negotiation.cc:769] Sending connection context
0927 14:57:13.769303 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 14:57:13.769690 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":143,"threads_started":1}
I20260927 14:57:13.771953 6423 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46547
I20260927 14:57:13.772215 6423 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:13.772749 6423 heartbeater.cc:511] Master 127.0.98.62:46547 requested a full tablet report, sending...
I20260927 14:57:13.773979 5964 ts_manager.cc:194] Registered new tserver with Master: 6aa67fb534924f2fb84ca241158b4477 (127.0.98.3:34577)
I20260927 14:57:13.774606 5964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48517
I20260927 14:57:13.778613 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/test-admin.keytab.
Sep 27 14:57:13 dist-test-slave-pz0h krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:13.795601 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 14:57:13.798342 6430 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:13.796999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45054 (local address 127.0.98.1:41401)
0927 14:57:13.797323 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:13.797330 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:13.797353 (+ 23us) server_negotiation.cc:407] Connection header received
0927 14:57:13.797399 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:13.797403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:13.797604 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:13.797733 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:13.798196 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45054: BlockingRecv error: recv got EOF from 127.0.0.1:45054 (error 108)
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":127,"threads_started":1}
W20260927 14:57:13.798527 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 14:57:13 dist-test-slave-pz0h krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:13.799727 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 14:57:13 dist-test-slave-pz0h krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:13.800802 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 14:57:13 dist-test-slave-pz0h krb5kdc[5919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 14:57:13.812361 6430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:13.801794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45056 (local address 127.0.98.1:41401)
0927 14:57:13.801921 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:13.801925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:13.801940 (+ 15us) server_negotiation.cc:407] Connection header received
0927 14:57:13.802089 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:13.802093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:13.802136 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:13.802198 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:13.802759 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:13.803611 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:13.804479 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:13.804900 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:13.805012 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:13.807778 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:13.807790 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:13.807806 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:13.807839 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:13.809849 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:13.810342 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:13.810347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:13.810349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:13.810438 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:13.810698 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:13.810702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:13.810703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:13.811237 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:13.811625 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:13.811979 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 14:57:13.812197 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 14:57:13.816813 6430 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:13.815801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45062 (local address 127.0.98.1:41401)
0927 14:57:13.815928 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:13.815931 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:13.816043 (+ 112us) server_negotiation.cc:407] Connection header received
0927 14:57:13.816168 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:13.816170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:13.816213 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:13.816271 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:13.816698 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45062: BlockingRecv error: recv got EOF from 127.0.0.1:45062 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 14:57:13.816973 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 14:57:13 dist-test-slave-pz0h krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:13.817957 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 14:57:13 dist-test-slave-pz0h krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:13.818763 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 14:57:13 dist-test-slave-pz0h krb5kdc[5919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 14:57:13.827315 6430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:13.819701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45074 (local address 127.0.98.1:41401)
0927 14:57:13.819828 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:13.819831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:13.819860 (+ 29us) server_negotiation.cc:407] Connection header received
0927 14:57:13.820016 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:13.820019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:13.820063 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:13.820136 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:13.820590 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:13.821132 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:13.822019 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:13.822261 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:13.822368 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:13.824562 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:13.824566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:13.824568 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:13.824595 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:13.825756 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:13.826179 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:13.826182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:13.826183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:13.826266 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:13.826510 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:13.826513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:13.826514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:13.826702 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:13.826886 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:13.827044 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 14:57:13.827156 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 14:57:13.829937 6430 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:13.829025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45090 (local address 127.0.98.1:41401)
0927 14:57:13.829146 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:13.829149 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:13.829175 (+ 26us) server_negotiation.cc:407] Connection header received
0927 14:57:13.829280 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:13.829282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:13.829324 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:13.829399 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:13.829827 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45090: BlockingRecv error: recv got EOF from 127.0.0.1:45090 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 14:57:13.829980 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790521004237705-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 14:57:13 dist-test-slave-pz0h krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:13.831002 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 14:57:13 dist-test-slave-pz0h krb5kdc[5919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:13.831836 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 14:57:13 dist-test-slave-pz0h krb5kdc[5919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 14:57:13.840088 6430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:13.832717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45098 (local address 127.0.98.1:41401)
0927 14:57:13.832871 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:13.832874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:13.832891 (+ 17us) server_negotiation.cc:407] Connection header received
0927 14:57:13.833009 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:13.833012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:13.833052 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:13.833119 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:13.833565 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:13.834080 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:13.834874 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:13.835098 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:13.835183 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:13.837317 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:13.837321 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:13.837323 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:13.837348 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:13.838661 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:13.839105 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:13.839108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:13.839110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:13.839189 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:13.839393 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:13.839395 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:13.839396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:13.839560 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:13.839670 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:13.839830 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 14:57:13.839930 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 14:57:13.841102 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 6026
I20260927 14:57:13.846709 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 6160
I20260927 14:57:13.851478 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 6294
I20260927 14:57:13.856436 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 5932
2026-09-27T14:57:13Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2158 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:57:13 dist-test-slave-pz0h krb5kdc[6443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:57:13 dist-test-slave-pz0h krb5kdc[6443](info): set up 2 sockets
Sep 27 14:57:13 dist-test-slave-pz0h krb5kdc[6443](info): commencing operation
krb5kdc: starting...
W20260927 14:57:14.815094 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 14:57:14 dist-test-slave-pz0h krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:57:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:57:14Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 14:57:14.937366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44545
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:15.048735 6456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:15.049046 6456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:15.049113 6456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:15.053407 6456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:57:15.053486 6456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:15.053514 6456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:15.053536 6456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:57:15.053557 6456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:57:15.059314 6456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:15.060503 6456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:15.061566 6456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:15.070245 6461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:15.070245 6464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:15.070341 6456 server_base.cc:1034] running on GCE node
W20260927 14:57:15.070250 6462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:15.071123 6456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:15.072273 6456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:15.073444 6456 hybrid_clock.cc:648] HybridClock initialized: now 1790521035073419 us; error 37 us; skew 500 ppm
Sep 27 14:57:15 dist-test-slave-pz0h krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521035, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:15.076356 6456 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 14:57:15.077333 6456 webserver.cc:460] Webserver started at http://127.0.98.62:32869/ using document root <none> and password file <none>
I20260927 14:57:15.077935 6456 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:15.078011 6456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:15.078226 6456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:15.080622 6456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "3e1e4b37da644604875d93c9bf1060ba"
format_stamp: "Formatted at 2026-09-27 14:57:15 on dist-test-slave-pz0h"
server_key: "32894d5077f5f0b857d541e04dace4ec"
server_key_iv: "e2285e30caf59b85497dfdcda8e73e3e"
server_key_version: "encryptionkey@0"
I20260927 14:57:15.081238 6456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "3e1e4b37da644604875d93c9bf1060ba"
format_stamp: "Formatted at 2026-09-27 14:57:15 on dist-test-slave-pz0h"
server_key: "32894d5077f5f0b857d541e04dace4ec"
server_key_iv: "e2285e30caf59b85497dfdcda8e73e3e"
server_key_version: "encryptionkey@0"
I20260927 14:57:15.085500 6456 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260927 14:57:15.088323 6471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:15.089783 6456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 14:57:15.089938 6456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "3e1e4b37da644604875d93c9bf1060ba"
format_stamp: "Formatted at 2026-09-27 14:57:15 on dist-test-slave-pz0h"
server_key: "32894d5077f5f0b857d541e04dace4ec"
server_key_iv: "e2285e30caf59b85497dfdcda8e73e3e"
server_key_version: "encryptionkey@0"
I20260927 14:57:15.090061 6456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:15.106739 6456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:15.110637 6456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:15.110888 6456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:15.118244 6456 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44545
I20260927 14:57:15.118515 6523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44545 every 8 connection(s)
I20260927 14:57:15.119342 6456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:57:15.122303 6524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:15.124344 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 6456
I20260927 14:57:15.124485 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 14:57:15.124742 392 external_mini_cluster.cc:1428] Setting key 18a3677a5ddfda927dff6bca6786cec6
I20260927 14:57:15.129755 6524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba: Bootstrap starting.
I20260927 14:57:15.132443 6524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:15.133451 6524 log.cc:824] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba: Log is configured to *not* fsync() on all Append() calls
Sep 27 14:57:15 dist-test-slave-pz0h krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 14:57:15.135901 6524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba: No bootstrap required, opened a new log
I20260927 14:57:15.139708 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:15.126319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56684 (local address 127.0.98.62:44545)
0927 14:57:15.126871 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:15.126888 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:15.126931 (+ 43us) server_negotiation.cc:407] Connection header received
0927 14:57:15.127818 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:15.127850 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:15.128276 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:15.128654 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:15.129517 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:15.130588 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:15.131503 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:15.132008 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:15.132174 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:15.135077 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:15.135082 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:15.135098 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:15.135127 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:15.136747 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:15.137213 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:15.137218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:15.137224 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:15.137341 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:15.137602 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:15.137605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:15.137606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:15.138044 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:15.138248 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:15.138530 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 14:57:15.138887 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":140,"threads_started":1}
I20260927 14:57:15.139642 6524 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } }
I20260927 14:57:15.139936 6524 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:15.139986 6524 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e1e4b37da644604875d93c9bf1060ba, State: Initialized, Role: FOLLOWER
I20260927 14:57:15.140466 6524 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } }
I20260927 14:57:15.140625 6524 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:57:15.140674 6524 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:57:15.140755 6524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:15.141775 6524 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } }
I20260927 14:57:15.142119 6524 leader_election.cc:302] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e1e4b37da644604875d93c9bf1060ba; no voters:
I20260927 14:57:15.142428 6524 leader_election.cc:288] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:57:15.142534 6529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:57:15.142737 6529 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [term 1 LEADER]: Becoming Leader. State: Replica: 3e1e4b37da644604875d93c9bf1060ba, State: Running, Role: LEADER
I20260927 14:57:15.143069 6529 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } }
I20260927 14:57:15.143654 6524 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:57:15.144665 6530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e1e4b37da644604875d93c9bf1060ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } }
I20260927 14:57:15.144668 6531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e1e4b37da644604875d93c9bf1060ba. Latest consensus state: current_term: 1 leader_uuid: "3e1e4b37da644604875d93c9bf1060ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } }
I20260927 14:57:15.145292 6531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:15.145361 6530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [sys.catalog]: This master's current role is: LEADER
W20260927 14:57:15.148672 6544 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:57:15.148756 6544 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:57:15.148895 6543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:57:15.149852 6543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:57:15.156203 6543 catalog_manager.cc:1347] Generated new cluster ID: 2dae8741e60142f9ba09d6c7b3a0d359
I20260927 14:57:15.156308 6543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:57:15.169462 6543 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:57:15.170859 6543 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:57:15.176317 6543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba: Generated new TSK 0
I20260927 14:57:15.177181 6543 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 14:57:15.214063 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44545
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:15.366679 6550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:15.367005 6550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:15.367056 6550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:15.371259 6550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:15.371333 6550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:15.371409 6550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:57:15.377032 6550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:15.378188 6550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:15.379285 6550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:15.389278 6555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:15.389278 6556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:15.389424 6550 server_base.cc:1034] running on GCE node
W20260927 14:57:15.389623 6558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:15.390069 6550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:15.390748 6550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:15.391963 6550 hybrid_clock.cc:648] HybridClock initialized: now 1790521035391941 us; error 41 us; skew 500 ppm
Sep 27 14:57:15 dist-test-slave-pz0h krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521035, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:15.395015 6550 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 14:57:15.396054 6550 webserver.cc:460] Webserver started at http://127.0.98.1:40249/ using document root <none> and password file <none>
I20260927 14:57:15.396651 6550 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:15.396723 6550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:15.396934 6550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:15.399190 6550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/data/instance:
uuid: "1544dca4984d47c494bb602d886ce11e"
format_stamp: "Formatted at 2026-09-27 14:57:15 on dist-test-slave-pz0h"
server_key: "126938ac4ca59442bc4c63192ad7495e"
server_key_iv: "98bd2fdd956366e361ffd9405f379077"
server_key_version: "encryptionkey@0"
I20260927 14:57:15.399817 6550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1544dca4984d47c494bb602d886ce11e"
format_stamp: "Formatted at 2026-09-27 14:57:15 on dist-test-slave-pz0h"
server_key: "126938ac4ca59442bc4c63192ad7495e"
server_key_iv: "98bd2fdd956366e361ffd9405f379077"
server_key_version: "encryptionkey@0"
I20260927 14:57:15.403999 6550 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 14:57:15.406629 6565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:15.408083 6550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:57:15.408232 6550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "1544dca4984d47c494bb602d886ce11e"
format_stamp: "Formatted at 2026-09-27 14:57:15 on dist-test-slave-pz0h"
server_key: "126938ac4ca59442bc4c63192ad7495e"
server_key_iv: "98bd2fdd956366e361ffd9405f379077"
server_key_version: "encryptionkey@0"
I20260927 14:57:15.408357 6550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:15.424790 6550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:15.427977 6550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:15.428185 6550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:15.428835 6550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:15.429708 6550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:15.429765 6550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:15.429822 6550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:15.429847 6550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:15.438760 6550 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34875
I20260927 14:57:15.438813 6678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34875 every 8 connection(s)
I20260927 14:57:15.439778 6550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:57:15.442624 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 6550
I20260927 14:57:15.442828 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:57:15.443585 392 external_mini_cluster.cc:1428] Setting key 38431286668fbe689666493300fd6374
Sep 27 14:57:15 dist-test-slave-pz0h krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521035, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260927 14:57:15.457444 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:15.442461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35785 (local address 127.0.98.62:44545)
0927 14:57:15.442673 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:15.442677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:15.444317 (+ 1640us) server_negotiation.cc:407] Connection header received
0927 14:57:15.446159 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:15.446163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:15.446221 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:15.446295 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:15.447678 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:15.448206 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:15.449804 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:15.450115 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:15.450215 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:15.453350 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:15.453353 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:15.453355 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:15.453385 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:15.454763 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:15.455352 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:15.455355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:15.455356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:15.455437 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:15.455772 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:15.455775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:15.455776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:15.455979 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:15.456058 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:15.456821 (+ 763us) server_negotiation.cc:299] Negotiation successful
0927 14:57:15.457125 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260927 14:57:15.457926 6681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:15.443007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.98.1:35785)
0927 14:57:15.444085 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:15.444161 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:15.445922 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:15.446494 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:15.446503 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:15.446977 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:15.447519 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:15.447533 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:15.448339 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:15.448343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:15.449659 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:15.449671 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:15.450304 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:15.450308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:15.450525 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:15.451193 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:15.451218 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:15.453077 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:15.454934 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:15.454940 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:15.454957 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:15.455254 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:15.455539 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:15.455541 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:15.455542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:15.455692 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:15.456068 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:15.456076 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:15.456468 (+ 392us) client_negotiation.cc:769] Sending connection context
0927 14:57:15.456655 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 14:57:15.456990 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"thread_start_us":396,"threads_started":1}
I20260927 14:57:15.459438 6679 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44545
I20260927 14:57:15.459721 6679 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:15.460318 6679 heartbeater.cc:511] Master 127.0.98.62:44545 requested a full tablet report, sending...
I20260927 14:57:15.461994 6488 ts_manager.cc:194] Registered new tserver with Master: 1544dca4984d47c494bb602d886ce11e (127.0.98.1:34875)
I20260927 14:57:15.463234 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35785
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 14:57:15.471371 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44545
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:15.587072 6684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:15.587353 6684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:15.587421 6684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:15.591600 6684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:15.591687 6684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:15.591771 6684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:57:15.597589 6684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:15.598731 6684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:15.599859 6684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:15.609977 6692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:15.609977 6689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:15.609977 6690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:15.610227 6684 server_base.cc:1034] running on GCE node
I20260927 14:57:15.610731 6684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:15.611449 6684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:15.612612 6684 hybrid_clock.cc:648] HybridClock initialized: now 1790521035612588 us; error 38 us; skew 500 ppm
Sep 27 14:57:15 dist-test-slave-pz0h krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521035, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:15.615568 6684 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 14:57:15.616497 6684 webserver.cc:460] Webserver started at http://127.0.98.2:38005/ using document root <none> and password file <none>
I20260927 14:57:15.617038 6684 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:15.617100 6684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:15.617326 6684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:15.619479 6684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/data/instance:
uuid: "5e76f2b076e14fcca79b8699d6be4704"
format_stamp: "Formatted at 2026-09-27 14:57:15 on dist-test-slave-pz0h"
server_key: "bb97c4f6727f7a81d7dbfed928679e29"
server_key_iv: "a3ef76829e051e87261505c238f58c4f"
server_key_version: "encryptionkey@0"
I20260927 14:57:15.620008 6684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5e76f2b076e14fcca79b8699d6be4704"
format_stamp: "Formatted at 2026-09-27 14:57:15 on dist-test-slave-pz0h"
server_key: "bb97c4f6727f7a81d7dbfed928679e29"
server_key_iv: "a3ef76829e051e87261505c238f58c4f"
server_key_version: "encryptionkey@0"
I20260927 14:57:15.623911 6684 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 14:57:15.626433 6699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:15.627852 6684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:57:15.627980 6684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "5e76f2b076e14fcca79b8699d6be4704"
format_stamp: "Formatted at 2026-09-27 14:57:15 on dist-test-slave-pz0h"
server_key: "bb97c4f6727f7a81d7dbfed928679e29"
server_key_iv: "a3ef76829e051e87261505c238f58c4f"
server_key_version: "encryptionkey@0"
I20260927 14:57:15.628084 6684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:15.664117 6684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:15.667554 6684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:15.667728 6684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:15.668370 6684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:15.669219 6684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:15.669267 6684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:15.669310 6684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:15.669339 6684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:15.677870 6684 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35207
I20260927 14:57:15.677899 6812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35207 every 8 connection(s)
I20260927 14:57:15.678879 6684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 14:57:15 dist-test-slave-pz0h krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521035, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 14:57:15.688442 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 6684
I20260927 14:57:15.688531 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:57:15.688781 392 external_mini_cluster.cc:1428] Setting key 91bdeedc585550abfdf1d4f3024db403
I20260927 14:57:15.692641 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:15.680657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51327 (local address 127.0.98.62:44545)
0927 14:57:15.680815 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:15.680820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:15.681701 (+ 881us) server_negotiation.cc:407] Connection header received
0927 14:57:15.683046 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:15.683049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:15.683099 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:15.683172 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:15.684304 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:15.684821 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:15.685951 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:15.686169 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:15.686251 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:15.688797 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:15.688800 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:15.688802 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:15.688829 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:15.690402 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:15.691049 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:15.691053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:15.691055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:15.691151 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:15.691482 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:15.691485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:15.691486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:15.691706 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:15.691794 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:15.692333 (+ 539us) server_negotiation.cc:299] Negotiation successful
0927 14:57:15.692467 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 14:57:15.693464 6815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:15.681078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.98.2:51327)
0927 14:57:15.681546 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:15.681589 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:15.682825 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:15.683329 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:15.683336 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:15.683704 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:15.684190 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:15.684204 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:15.684939 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:15.684942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:15.685826 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:15.685835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:15.686293 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:15.686296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:15.686448 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:15.687098 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:15.687115 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:15.688613 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:15.690544 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:15.690550 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:15.690567 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:15.690945 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:15.691240 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:15.691242 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:15.691243 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:15.691399 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:15.691793 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:15.691799 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:15.692075 (+ 276us) client_negotiation.cc:769] Sending connection context
0927 14:57:15.692250 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 14:57:15.692623 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":117,"threads_started":1}
I20260927 14:57:15.694589 6813 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44545
I20260927 14:57:15.694902 6813 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:15.695571 6813 heartbeater.cc:511] Master 127.0.98.62:44545 requested a full tablet report, sending...
I20260927 14:57:15.696729 6488 ts_manager.cc:194] Registered new tserver with Master: 5e76f2b076e14fcca79b8699d6be4704 (127.0.98.2:35207)
I20260927 14:57:15.697305 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51327
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 14:57:15.713851 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44545
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:15.830291 6818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:15.830610 6818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:15.830657 6818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:15.835400 6818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:15.835489 6818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:15.835554 6818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:57:15.841403 6818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:15.842605 6818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:15.843768 6818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:15.854306 6823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:15.854306 6824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:15.854643 6818 server_base.cc:1034] running on GCE node
W20260927 14:57:15.854856 6826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:15.855437 6818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:15.856376 6818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:15.857587 6818 hybrid_clock.cc:648] HybridClock initialized: now 1790521035857562 us; error 49 us; skew 500 ppm
Sep 27 14:57:15 dist-test-slave-pz0h krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521035, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:15.861737 6818 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 14:57:15.862908 6818 webserver.cc:460] Webserver started at http://127.0.98.3:33941/ using document root <none> and password file <none>
I20260927 14:57:15.863651 6818 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:15.863742 6818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:15.864023 6818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:15.866250 6818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/data/instance:
uuid: "280b4c9e2726475989c4e06525b89aec"
format_stamp: "Formatted at 2026-09-27 14:57:15 on dist-test-slave-pz0h"
server_key: "7df4b743ca0ca3a4bfb230eeb6b4c0a3"
server_key_iv: "27ba8019dd30abbfce5c09cc4c53f6ac"
server_key_version: "encryptionkey@0"
I20260927 14:57:15.866815 6818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "280b4c9e2726475989c4e06525b89aec"
format_stamp: "Formatted at 2026-09-27 14:57:15 on dist-test-slave-pz0h"
server_key: "7df4b743ca0ca3a4bfb230eeb6b4c0a3"
server_key_iv: "27ba8019dd30abbfce5c09cc4c53f6ac"
server_key_version: "encryptionkey@0"
I20260927 14:57:15.870752 6818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 14:57:15.873704 6833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:15.875279 6818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:57:15.875417 6818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "280b4c9e2726475989c4e06525b89aec"
format_stamp: "Formatted at 2026-09-27 14:57:15 on dist-test-slave-pz0h"
server_key: "7df4b743ca0ca3a4bfb230eeb6b4c0a3"
server_key_iv: "27ba8019dd30abbfce5c09cc4c53f6ac"
server_key_version: "encryptionkey@0"
I20260927 14:57:15.875536 6818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:15.911367 6818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:15.915455 6818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:15.915664 6818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:15.916299 6818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:15.917201 6818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:15.917249 6818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:15.917290 6818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:15.917304 6818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:15.926189 6818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42813
I20260927 14:57:15.926362 6946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42813 every 8 connection(s)
I20260927 14:57:15.927255 6818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:57:15.931113 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 6818
I20260927 14:57:15.931241 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:57:15.931546 392 external_mini_cluster.cc:1428] Setting key 57de9d69e026898e95981ac49c9eea89
Sep 27 14:57:15 dist-test-slave-pz0h krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521035, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 14:57:15.942260 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:15.929230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49715 (local address 127.0.98.62:44545)
0927 14:57:15.929355 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:15.929358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:15.930316 (+ 958us) server_negotiation.cc:407] Connection header received
0927 14:57:15.932030 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:15.932034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:15.932098 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:15.932174 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:15.933850 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:15.934384 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:15.935727 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:15.935958 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:15.936049 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:15.938766 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:15.938770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:15.938772 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:15.938824 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:15.940145 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:15.940709 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:15.940713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:15.940714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:15.940790 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:15.941123 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:15.941126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:15.941128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:15.941320 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:15.941410 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:15.941960 (+ 550us) server_negotiation.cc:299] Negotiation successful
0927 14:57:15.942090 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 14:57:15.943207 6949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:15.929549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.98.3:49715)
0927 14:57:15.930125 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:15.930173 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:15.931765 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:15.932360 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:15.932368 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:15.932879 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:15.933603 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:15.933626 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:15.934525 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:15.934529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:15.935591 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:15.935603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:15.936115 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:15.936119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:15.936304 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:15.936936 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:15.936954 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:15.938538 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:15.940288 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:15.940294 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:15.940311 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:15.940604 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:15.940878 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:15.940881 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:15.940883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:15.941031 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:15.941435 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:15.941442 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:15.941711 (+ 269us) client_negotiation.cc:769] Sending connection context
0927 14:57:15.941892 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 14:57:15.942254 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":142,"threads_started":1}
I20260927 14:57:15.944960 6947 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44545
I20260927 14:57:15.945245 6947 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:15.945896 6947 heartbeater.cc:511] Master 127.0.98.62:44545 requested a full tablet report, sending...
I20260927 14:57:15.947039 6488 ts_manager.cc:194] Registered new tserver with Master: 280b4c9e2726475989c4e06525b89aec (127.0.98.3:42813)
I20260927 14:57:15.947628 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49715
I20260927 14:57:15.958040 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260927 14:57:15.984114 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36483
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44545,127.0.98.61:36483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:16.101624 6952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:16.101923 6952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:16.101972 6952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:16.106276 6952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:57:16.106364 6952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:16.106381 6952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:16.106397 6952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:57:16.106411 6952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:57:16.112434 6952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44545,127.0.98.61:36483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
W20260927 14:57:16.112711 6952 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:44545,127.0.98.61:36483'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 14:57:16.113647 6952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:16.114705 6952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:16.124714 6958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:16.124699 6957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:16.125069 6952 server_base.cc:1034] running on GCE node
W20260927 14:57:16.124728 6960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:16.125736 6952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:16.127018 6952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:16.128185 6952 hybrid_clock.cc:648] HybridClock initialized: now 1790521036128159 us; error 52 us; skew 500 ppm
Sep 27 14:57:16 dist-test-slave-pz0h krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521036, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:16.131184 6952 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 14:57:16.132220 6952 webserver.cc:460] Webserver started at http://127.0.98.61:45823/ using document root <none> and password file <none>
I20260927 14:57:16.132773 6952 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:16.132834 6952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:16.133036 6952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:16.135257 6952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-1/data/instance:
uuid: "86c641e5060c4fe88d0b39cf0550cd6f"
format_stamp: "Formatted at 2026-09-27 14:57:16 on dist-test-slave-pz0h"
server_key: "3a5735c2a14ab72641f8d899e60cfac6"
server_key_iv: "8c654aaf33336a29956635c163068b06"
server_key_version: "encryptionkey@0"
I20260927 14:57:16.135807 6952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-1/wal/instance:
uuid: "86c641e5060c4fe88d0b39cf0550cd6f"
format_stamp: "Formatted at 2026-09-27 14:57:16 on dist-test-slave-pz0h"
server_key: "3a5735c2a14ab72641f8d899e60cfac6"
server_key_iv: "8c654aaf33336a29956635c163068b06"
server_key_version: "encryptionkey@0"
I20260927 14:57:16.139575 6952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 14:57:16.142158 6967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:16.143496 6952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 14:57:16.143623 6952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-1/wal
uuid: "86c641e5060c4fe88d0b39cf0550cd6f"
format_stamp: "Formatted at 2026-09-27 14:57:16 on dist-test-slave-pz0h"
server_key: "3a5735c2a14ab72641f8d899e60cfac6"
server_key_iv: "8c654aaf33336a29956635c163068b06"
server_key_version: "encryptionkey@0"
I20260927 14:57:16.143730 6952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:16.175149 6952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:16.178305 6952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:16.178490 6952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:16.185537 6952 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36483
I20260927 14:57:16.185801 7019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36483 every 8 connection(s)
I20260927 14:57:16.186604 6952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-1/data/info.pb
I20260927 14:57:16.189460 7020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:16.191087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 6952
I20260927 14:57:16.193563 7020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 14:57:16 dist-test-slave-pz0h krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 14:57:16 dist-test-slave-pz0h krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521036, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 14:57:16.204811 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.195600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56694 (local address 127.0.98.62:44545)
0927 14:57:16.195723 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:16.195726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:16.196061 (+ 335us) server_negotiation.cc:407] Connection header received
0927 14:57:16.196182 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:16.196185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:16.196229 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:16.196295 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:16.196930 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.197445 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.198412 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.198629 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.198708 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.201242 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:16.201246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:16.201248 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:16.201279 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:16.202836 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.203333 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.203336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.203337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.203413 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.203728 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.203731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.203733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.204006 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:16.204111 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:16.204498 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 14:57:16.204641 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 14:57:16.205705 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.192599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42698 (local address 127.0.98.61:36483)
0927 14:57:16.193013 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:16.193028 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:16.193064 (+ 36us) server_negotiation.cc:407] Connection header received
0927 14:57:16.193760 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:16.193779 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:16.194205 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:16.194516 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:16.195412 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.196774 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.197672 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.198167 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.198291 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.200812 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:16.200817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:16.200829 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:16.200852 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:16.202762 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.203266 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.203271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.203272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.203408 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.203695 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.203698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.203700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.204182 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:16.204321 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:16.204569 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 14:57:16.204786 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":269,"thread_start_us":121,"threads_started":1}
I20260927 14:57:16.205704 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.195563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.0.1:56694)
0927 14:57:16.195944 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:16.195987 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:16.196097 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:16.196350 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:16.196355 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:16.196687 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:16.196803 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.196809 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.197562 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.197565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.198280 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.198289 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.198773 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.198778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.198913 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.199362 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:16.199377 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:16.201091 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:16.202950 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.202959 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.202970 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.203247 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.203496 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.203499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.203501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.203649 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.204117 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:16.204123 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:16.204255 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 14:57:16.204444 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 14:57:16.204897 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":134,"thread_start_us":154,"threads_started":1}
I20260927 14:57:16.207172 7020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 14:57:16 dist-test-slave-pz0h krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521036, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 14:57:16.214499 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.207743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36483 (local address 127.0.0.1:42714)
0927 14:57:16.207850 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:16.207864 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:16.207946 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:16.208138 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:16.208141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:16.208289 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:16.208386 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.208391 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.209435 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.209437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.209950 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.209957 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.210327 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.210329 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.210395 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.210854 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:16.210865 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:16.212047 (+ 1182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:16.213398 (+ 1351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.213400 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.213402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.213569 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.213766 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.213768 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.213770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.213830 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.214105 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:16.214107 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:16.214175 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 14:57:16.214243 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 14:57:16.214351 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 14:57:16.214524 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.207814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42714 (local address 127.0.98.61:36483)
0927 14:57:16.207897 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:16.207901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:16.207923 (+ 22us) server_negotiation.cc:407] Connection header received
0927 14:57:16.208014 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:16.208017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:16.208053 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:16.208119 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:16.208494 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.209338 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.210038 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.210250 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.210321 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.212157 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:16.212160 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:16.212161 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:16.212181 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:16.213312 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.213647 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.213650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.213651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.213709 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.213884 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.213886 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.213887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.214045 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:16.214111 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:16.214281 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 14:57:16.214403 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":33}
I20260927 14:57:16.217231 7020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f: Bootstrap starting.
I20260927 14:57:16.219559 7020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:16.220289 7020 log.cc:824] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f: Log is configured to *not* fsync() on all Append() calls
I20260927 14:57:16.222227 7020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f: No bootstrap required, opened a new log
I20260927 14:57:16.225253 7020 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } }
I20260927 14:57:16.225482 7020 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:16.225517 7020 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86c641e5060c4fe88d0b39cf0550cd6f, State: Initialized, Role: FOLLOWER
I20260927 14:57:16.225968 7020 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } }
I20260927 14:57:16.226451 7026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } } }
I20260927 14:57:16.226583 7026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:57:16.227210 7020 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [sys.catalog]: configured and running, proceeding with master startup.
W20260927 14:57:16.231928 7037 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:57:16.232038 7037 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:57:16.234313 6952 master_runner.cc:428] Detected that this master 86c641e5060c4fe88d0b39cf0550cd6f is joining an existing cluster
I20260927 14:57:16.234400 6952 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:36483
I20260927 14:57:16.245015 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.236584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56708 (local address 127.0.98.62:44545)
0927 14:57:16.236733 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:16.236737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:16.237059 (+ 322us) server_negotiation.cc:407] Connection header received
0927 14:57:16.237178 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:16.237181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:16.237236 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:16.237322 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:16.238233 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.239068 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.240446 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.240672 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.240765 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.241803 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:16.241806 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:16.241808 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:16.241833 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:16.243276 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.243801 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.243805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.243807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.243899 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.244165 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.244169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.244170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.244372 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:16.244523 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:16.244730 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 14:57:16.244871 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 14:57:16.245031 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.236494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.0.1:56708)
0927 14:57:16.236960 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:16.236975 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:16.237091 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:16.237807 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:16.237810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:16.237998 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:16.238109 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.238115 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.239185 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.239188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.240299 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.240308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.240784 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.240787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.240896 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.241325 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:16.241338 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:16.241678 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:16.243382 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.243386 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.243388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.243699 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.243986 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.243988 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.243989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.244071 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.244472 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:16.244475 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:16.244575 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 14:57:16.244666 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 14:57:16.244822 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":1,"thread_start_us":116,"threads_started":1}
I20260927 14:57:16.245630 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.236853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42720 (local address 127.0.98.61:36483)
0927 14:57:16.236945 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:16.236949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:16.237084 (+ 135us) server_negotiation.cc:407] Connection header received
0927 14:57:16.237213 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:16.237216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:16.237250 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:16.237312 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:16.237791 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.238443 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.239275 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.239660 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.239764 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.241459 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:16.241463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:16.241466 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:16.241492 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:16.243362 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.244016 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.244041 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.244043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.244129 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.244436 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.244439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.244441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.244628 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:16.244693 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:16.244955 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 14:57:16.245473 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":29}
I20260927 14:57:16.246124 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.236516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36483 (local address 127.0.0.1:42720)
0927 14:57:16.237025 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:16.237036 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:16.237122 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:16.237356 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:16.237358 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:16.237581 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:16.237680 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.237685 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.238530 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.238533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.239121 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.239130 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.239791 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.239795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.239889 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.240912 (+ 1023us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:16.240926 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:16.241319 (+ 393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:16.243448 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.243451 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.243452 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.243699 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.244254 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.244258 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.244259 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.244336 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.244746 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:16.244749 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:16.244819 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 14:57:16.245813 (+ 994us) client_negotiation.cc:240] Negotiation successful
0927 14:57:16.245951 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":112,"threads_started":1}
Sep 27 14:57:16 dist-test-slave-pz0h krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521035, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 14:57:16.259342 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.249819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42732 (local address 127.0.98.61:36483)
0927 14:57:16.249940 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:16.249944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:16.250453 (+ 509us) server_negotiation.cc:407] Connection header received
0927 14:57:16.250522 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:16.250526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:16.250579 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:16.250666 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:16.251358 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.252154 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.252998 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.253344 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.253445 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.255970 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:16.255974 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:16.255977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:16.256007 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:16.257498 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.257996 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.258000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.258001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.258092 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.258333 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.258336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.258338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.258553 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:16.258644 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:16.259051 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 14:57:16.259175 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:57:16.259342 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.249768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36483 (local address 127.0.0.1:42732)
0927 14:57:16.250126 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:16.250175 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:16.250313 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:16.250739 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:16.250744 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:16.251132 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:16.251251 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.251257 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.252248 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.252250 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.252895 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.252903 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.253483 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.253486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.253578 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.254142 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:16.254154 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:16.255800 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:16.257629 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.257639 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.257645 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.257909 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.258169 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.258172 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.258173 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.258252 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.258651 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:16.258658 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:16.258830 (+ 172us) client_negotiation.cc:769] Sending connection context
0927 14:57:16.258983 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 14:57:16.259190 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":95,"threads_started":1}
I20260927 14:57:16.260959 6488 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:36483: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 36483
}
attrs {
promote: true
}
}
dest_uuid: "3e1e4b37da644604875d93c9bf1060ba"
cas_config_opid_index: -1
I20260927 14:57:16.261698 6488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: true } }
I20260927 14:57:16.264310 6994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:16.265493 6994 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [term 1 FOLLOWER]: Refusing update from remote peer 3e1e4b37da644604875d93c9bf1060ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 14:57:16.263932 7048 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 86c641e5060c4fe88d0b39cf0550cd6f (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: true } } }
I20260927 14:57:16.266940 7049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:57:16.267720 7048 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:36483
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 14:57:16.268424 6952 master.cc:537] Master@127.0.98.61:36483 shutting down...
I20260927 14:57:16.269029 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3e1e4b37da644604875d93c9bf1060ba" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: true } } }
I20260927 14:57:16.269155 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [sys.catalog]: This master's current role is: LEADER
W20260927 14:57:16.269490 6474 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 14:57:16.270649 6474 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba -> Peer 86c641e5060c4fe88d0b39cf0550cd6f (127.0.98.61:36483): Couldn't send request to peer 86c641e5060c4fe88d0b39cf0550cd6f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260927 14:57:16.272991 6952 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:57:16.273353 6952 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:57:16.273463 6952 tablet_replica.cc:330] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f: stopping tablet replica
I20260927 14:57:16.278429 6952 master.cc:552] Master@127.0.98.61:36483 shutdown complete.
I20260927 14:57:16.278501 6952 master_runner.cc:305] Clearing existing system tablet
I20260927 14:57:16.278841 6952 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:16.278904 6952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:16.280613 6952 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 14:57:16.281572 7055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:16.281775 6952 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:16.281885 6952 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 14:57:16.281924 6952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-1/wal
uuid: "86c641e5060c4fe88d0b39cf0550cd6f"
format_stamp: "Formatted at 2026-09-27 14:57:16 on dist-test-slave-pz0h"
server_key: "3a5735c2a14ab72641f8d899e60cfac6"
server_key_iv: "8c654aaf33336a29956635c163068b06"
server_key_version: "encryptionkey@0"
I20260927 14:57:16.284186 6952 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:57:16.286916 6952 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 14:57:16.287019 6952 log.cc:1196] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 14:57:16.287271 6952 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f: Deleting consensus metadata
I20260927 14:57:16.288089 6952 master_runner.cc:315] Copying system tablet from 127.0.98.62:44545
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260927 14:57:16.289289 6952 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:44545
I20260927 14:57:16.289474 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41351
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44545,127.0.98.61:36483,127.0.98.60:41351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:57:16.300235 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.291113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56720 (local address 127.0.98.62:44545)
0927 14:57:16.291251 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:16.291256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:16.291772 (+ 516us) server_negotiation.cc:407] Connection header received
0927 14:57:16.291910 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:16.291913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:16.291974 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:16.292046 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:16.292548 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.293329 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.294486 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.294912 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.295064 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.296607 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:16.296612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:16.296615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:16.296645 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:16.298238 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.298756 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.298760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.298762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.298884 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.299215 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.299218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.299220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.299455 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:16.299532 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:16.299906 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 14:57:16.300053 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 14:57:16.300836 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.291351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.0.1:56720)
0927 14:57:16.291658 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:16.291674 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:16.291805 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:16.292075 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:16.292078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:16.292308 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:16.292433 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.292440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.293436 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.293440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.294368 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.294379 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.295216 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.295220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.295405 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.295966 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:16.295981 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:16.296431 (+ 450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:16.298376 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.298379 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.298381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.298639 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.299017 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.299020 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.299023 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.299110 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.299605 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:16.299608 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:16.299714 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 14:57:16.300477 (+ 763us) client_negotiation.cc:240] Negotiation successful
0927 14:57:16.300643 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":111,"threads_started":1}
I20260927 14:57:16.301437 6508 tablet_copy_service.cc:140] P 3e1e4b37da644604875d93c9bf1060ba: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 86c641e5060c4fe88d0b39cf0550cd6f ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:56720)
I20260927 14:57:16.301637 6508 tablet_copy_service.cc:161] P 3e1e4b37da644604875d93c9bf1060ba: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 86c641e5060c4fe88d0b39cf0550cd6f at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:56720: session id = 86c641e5060c4fe88d0b39cf0550cd6f-00000000000000000000000000000000
I20260927 14:57:16.303586 6508 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 14:57:16.305403 6952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 14:57:16.308054 6952 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f: tablet copy: Starting download of 0 data blocks...
I20260927 14:57:16.308290 6952 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f: tablet copy: Starting download of 1 WAL segments...
I20260927 14:57:16.310405 6952 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 14:57:16.312793 6508 tablet_copy_service.cc:342] P 3e1e4b37da644604875d93c9bf1060ba: Request end of tablet copy session 86c641e5060c4fe88d0b39cf0550cd6f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:56720
I20260927 14:57:16.312953 6508 tablet_copy_service.cc:433] P 3e1e4b37da644604875d93c9bf1060ba: ending tablet copy session 86c641e5060c4fe88d0b39cf0550cd6f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 86c641e5060c4fe88d0b39cf0550cd6f
I20260927 14:57:16.320703 6952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:16.322328 7063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:16.322357 7064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:16.322468 7066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:16.322863 6952 server_base.cc:1034] running on GCE node
I20260927 14:57:16.323011 6952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:16.323297 6952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:16.324441 6952 hybrid_clock.cc:648] HybridClock initialized: now 1790521036324426 us; error 33 us; skew 500 ppm
Sep 27 14:57:16 dist-test-slave-pz0h krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521036, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:16.325920 6952 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 14:57:16.326368 6952 webserver.cc:460] Webserver started at http://127.0.98.61:37581/ using document root <none> and password file <none>
I20260927 14:57:16.326545 6952 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:16.326583 6952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:16.327855 6952 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 14:57:16.328822 7073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:16.329094 6952 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 14:57:16.329169 6952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-1/wal
uuid: "86c641e5060c4fe88d0b39cf0550cd6f"
format_stamp: "Formatted at 2026-09-27 14:57:16 on dist-test-slave-pz0h"
server_key: "3a5735c2a14ab72641f8d899e60cfac6"
server_key_iv: "8c654aaf33336a29956635c163068b06"
server_key_version: "encryptionkey@0"
I20260927 14:57:16.329247 6952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:16.335175 6952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:16.337200 6952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:16.341864 6952 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36483
I20260927 14:57:16.341892 7125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36483 every 8 connection(s)
I20260927 14:57:16.342466 6952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-1/data/info.pb
I20260927 14:57:16.344437 7126 sys_catalog.cc:263] Verifying existing consensus state
I20260927 14:57:16.345547 7126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f: Bootstrap starting.
I20260927 14:57:16.356496 7126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 14:57:16.356860 7126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f: Bootstrap complete.
I20260927 14:57:16.357513 7126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: true } }
I20260927 14:57:16.357656 7126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 86c641e5060c4fe88d0b39cf0550cd6f, State: Initialized, Role: LEARNER
I20260927 14:57:16.357753 7126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: true } }
I20260927 14:57:16.358119 7129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: true } } }
I20260927 14:57:16.358223 7126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:57:16.358325 7129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [sys.catalog]: This master's current role is: LEARNER
I20260927 14:57:16.361829 7140 catalog_manager.cc:1259] Loaded cluster ID: 2dae8741e60142f9ba09d6c7b3a0d359
I20260927 14:57:16.361927 7140 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f: loading cluster ID for follower catalog manager: success
I20260927 14:57:16.364377 7140 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f: acquiring CA information for follower catalog manager: success
I20260927 14:57:16.365971 7140 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 14:57:16.428937 7061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:16.429327 7061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:16.429402 7061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:16.434005 7061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:57:16.434091 7061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:16.434119 7061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:16.434142 7061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:57:16.434172 7061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:57:16.440048 7061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44545,127.0.98.61:36483,127.0.98.60:41351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:16.441147 7061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:16.442179 7061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:16.451717 7148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:16.452096 7145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:16.452190 7146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:16.452255 7061 server_base.cc:1034] running on GCE node
I20260927 14:57:16.452891 7061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:16.454147 7061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:16.455338 7061 hybrid_clock.cc:648] HybridClock initialized: now 1790521036455310 us; error 50 us; skew 500 ppm
Sep 27 14:57:16 dist-test-slave-pz0h krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521036, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:16.459041 7061 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 14:57:16.460139 7061 webserver.cc:460] Webserver started at http://127.0.98.60:36865/ using document root <none> and password file <none>
I20260927 14:57:16.460718 7061 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:16.460789 7061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:16.460999 7061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:16.463279 7061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-2/data/instance:
uuid: "56ffb5d4c52e4a5fa29276ae2214dc99"
format_stamp: "Formatted at 2026-09-27 14:57:16 on dist-test-slave-pz0h"
server_key: "45f7487fe34909a7bdb12d736aa610bc"
server_key_iv: "c6f475fdadc96eb47907c1a07cd00cbb"
server_key_version: "encryptionkey@0"
I20260927 14:57:16.463891 7061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-2/wal/instance:
uuid: "56ffb5d4c52e4a5fa29276ae2214dc99"
format_stamp: "Formatted at 2026-09-27 14:57:16 on dist-test-slave-pz0h"
server_key: "45f7487fe34909a7bdb12d736aa610bc"
server_key_iv: "c6f475fdadc96eb47907c1a07cd00cbb"
server_key_version: "encryptionkey@0"
I20260927 14:57:16.466013 6679 heartbeater.cc:503] Master 127.0.98.62:44545 was elected leader, sending a full tablet report...
I20260927 14:57:16.468225 7061 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 14:57:16.470984 7155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:16.472313 7061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:57:16.472446 7061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-2/wal
uuid: "56ffb5d4c52e4a5fa29276ae2214dc99"
format_stamp: "Formatted at 2026-09-27 14:57:16 on dist-test-slave-pz0h"
server_key: "45f7487fe34909a7bdb12d736aa610bc"
server_key_iv: "c6f475fdadc96eb47907c1a07cd00cbb"
server_key_version: "encryptionkey@0"
I20260927 14:57:16.472579 7061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:16.487671 7061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:16.490942 7061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:16.491132 7061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:16.499032 7061 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41351
I20260927 14:57:16.499397 7207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41351 every 8 connection(s)
I20260927 14:57:16.500183 7061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-2/data/info.pb
I20260927 14:57:16.503736 7208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:16.506923 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 7061
I20260927 14:57:16.507983 7208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 14:57:16 dist-test-slave-pz0h krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521036, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 14:57:16 dist-test-slave-pz0h krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 14:57:16.520759 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.510168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56724 (local address 127.0.98.62:44545)
0927 14:57:16.510292 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:16.510296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:16.510487 (+ 191us) server_negotiation.cc:407] Connection header received
0927 14:57:16.510640 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:16.510644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:16.510692 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:16.510740 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:16.511765 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.512281 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.513537 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.513961 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.514077 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.516702 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:16.516707 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:16.516710 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:16.516744 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:16.518255 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.518942 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.518946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.518947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.519037 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.519392 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.519396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.519398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.519642 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:16.519746 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:16.520422 (+ 676us) server_negotiation.cc:299] Negotiation successful
0927 14:57:16.520582 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 14:57:16.521817 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.508499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35580 (local address 127.0.98.60:41351)
0927 14:57:16.508946 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:16.508966 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:16.509009 (+ 43us) server_negotiation.cc:407] Connection header received
0927 14:57:16.509835 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:16.509859 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:16.510303 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:16.510663 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:16.511347 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.512462 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.513424 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.513893 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.514026 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.517146 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:16.517154 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:16.517170 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:16.517197 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:16.519021 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.519435 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.519440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.519441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.519507 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.519751 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.519754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.519755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.520119 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:16.520309 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:16.520572 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 14:57:16.520955 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":241,"server-negotiator.queue_time_us":296,"thread_start_us":128,"threads_started":1}
I20260927 14:57:16.521817 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.509961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.0.1:56724)
0927 14:57:16.510347 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:16.510395 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:16.510527 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:16.510943 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:16.510948 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:16.511350 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:16.511601 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.511614 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.512410 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.512414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.513404 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.513416 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.514123 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.514127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.514242 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.514831 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:16.514855 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:16.516492 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:16.518373 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.518381 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.518394 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.518837 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.519140 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.519143 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.519144 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.519287 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.519747 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:16.519754 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:16.520123 (+ 369us) client_negotiation.cc:769] Sending connection context
0927 14:57:16.520330 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 14:57:16.520721 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"spinlock_wait_cycles":768,"thread_start_us":121,"threads_started":1}
I20260927 14:57:16.524107 7208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 14:57:16 dist-test-slave-pz0h krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521036, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 14:57:16.532711 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.524636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36483 (local address 127.0.0.1:42738)
0927 14:57:16.524764 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:16.524779 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:16.524863 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:16.525219 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:16.525221 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:16.525372 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:16.525475 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.525480 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.526360 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.526363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.526990 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.527001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.527479 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.527483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.527560 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.528008 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:16.528019 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:16.529344 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:16.531182 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.531185 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.531188 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.531398 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.531688 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.531691 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.531694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.531770 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.532183 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:16.532186 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:16.532276 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 14:57:16.532364 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 14:57:16.532531 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 14:57:16.532713 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.524742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42738 (local address 127.0.98.61:36483)
0927 14:57:16.525009 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:16.525013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:16.525035 (+ 22us) server_negotiation.cc:407] Connection header received
0927 14:57:16.525073 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:16.525075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:16.525115 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:16.525196 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:16.525572 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.526259 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.527118 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.527360 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.527451 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.529508 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:16.529512 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:16.529514 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:16.529540 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:16.531053 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.531514 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.531517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.531518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.531601 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.531852 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.531855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.531857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.532093 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:16.532182 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:16.532425 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 14:57:16.532549 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":98,"threads_started":1}
I20260927 14:57:16.533520 7208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41351 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 14:57:16 dist-test-slave-pz0h krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521036, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 14:57:16.541081 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.533966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41351 (local address 127.0.0.1:35596)
0927 14:57:16.534065 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:16.534078 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:16.534142 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:16.534361 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:16.534364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:16.534505 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:16.534594 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.534599 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.535437 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.535440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.535943 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.535950 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.536394 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.536397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.536473 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.536908 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:16.536918 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:16.538171 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:16.539826 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.539829 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.539834 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.540039 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.540278 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.540280 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.540281 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.540346 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.540656 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:16.540658 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:16.540736 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 14:57:16.540815 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 14:57:16.540935 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 14:57:16.541107 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.534014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35596 (local address 127.0.98.60:41351)
0927 14:57:16.534123 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:16.534126 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:16.534137 (+ 11us) server_negotiation.cc:407] Connection header received
0927 14:57:16.534204 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:16.534207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:16.534243 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:16.534303 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:16.534730 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.535335 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.536039 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.536276 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.536354 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.538365 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:16.538368 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:16.538370 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:16.538392 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:16.539706 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.540132 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.540135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.540136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.540209 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.540404 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.540406 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.540407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.540584 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:16.540650 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:16.540861 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 14:57:16.540987 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":46}
I20260927 14:57:16.544495 7208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99: Bootstrap starting.
I20260927 14:57:16.547834 7208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:16.548638 7208 log.cc:824] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99: Log is configured to *not* fsync() on all Append() calls
I20260927 14:57:16.551420 7208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99: No bootstrap required, opened a new log
I20260927 14:57:16.554594 7208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41351 } }
I20260927 14:57:16.554879 7208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:16.554930 7208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56ffb5d4c52e4a5fa29276ae2214dc99, State: Initialized, Role: FOLLOWER
I20260927 14:57:16.555400 7208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41351 } }
I20260927 14:57:16.555891 7214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41351 } } }
I20260927 14:57:16.556030 7214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:57:16.556663 7208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 14:57:16.561434 7225 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:57:16.561545 7225 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:57:16.565963 7061 master_runner.cc:428] Detected that this master 56ffb5d4c52e4a5fa29276ae2214dc99 is joining an existing cluster
I20260927 14:57:16.566035 7061 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:41351
I20260927 14:57:16.575572 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 6550
I20260927 14:57:16.577872 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.568196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56734 (local address 127.0.98.62:44545)
0927 14:57:16.568321 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:16.568325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:16.569564 (+ 1239us) server_negotiation.cc:407] Connection header received
0927 14:57:16.569717 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:16.569720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:16.569770 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:16.569819 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:16.570479 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.571301 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.572777 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.573106 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.573180 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.574545 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:16.574548 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:16.574550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:16.574576 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:16.576171 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.576603 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.576607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.576609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.576695 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.576944 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.576948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.576950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.577152 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:16.577241 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:16.577502 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 14:57:16.577619 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 14:57:16.577898 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.568405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.0.1:56734)
0927 14:57:16.568963 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:16.568975 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:16.569625 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:16.569957 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:16.569960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:16.570235 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:16.570359 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.570365 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.571429 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.571431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.572647 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.572657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.573318 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.573321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.573411 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.574107 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:16.574119 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:16.574406 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:16.576266 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.576269 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.576273 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.576505 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.576778 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.576781 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.576783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.576865 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.577238 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:16.577241 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:16.577341 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 14:57:16.577441 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 14:57:16.577605 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":66,"thread_start_us":88,"threads_started":1}
I20260927 14:57:16.583408 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34875
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44545
--webserver_port=40249
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:44545,127.0.98.61:36483
--tserver_master_addrs=127.0.98.62:44545,127.0.98.61:36483,127.0.98.60:41351 with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 14:57:16 dist-test-slave-pz0h krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521035, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 14:57:16.598219 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.568142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41351 (local address 127.0.0.1:35610)
0927 14:57:16.568614 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:16.568628 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:16.568738 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:16.569030 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:16.569033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:16.569338 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:16.569500 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.569507 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.570394 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.570397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.571301 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.571310 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.572016 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.572019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.572104 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.572596 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:16.572610 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:16.573101 (+ 491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:16.596543 (+ 23442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.596547 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.596549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.596800 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.597091 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.597094 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.597097 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.597182 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.597648 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:16.597650 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:16.597754 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 14:57:16.597856 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 14:57:16.598007 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":109,"threads_started":1}
I20260927 14:57:16.598953 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.568308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35610 (local address 127.0.98.60:41351)
0927 14:57:16.568557 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:16.568561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:16.568707 (+ 146us) server_negotiation.cc:407] Connection header received
0927 14:57:16.568819 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:16.568822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:16.568861 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:16.568923 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:16.569597 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.570291 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.571409 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.571813 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.571894 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.573226 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:16.573230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:16.573232 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:16.573251 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:16.596382 (+ 23131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.596914 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.596918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.596919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.597006 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.597273 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.597275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.597277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.597509 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:16.597613 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:16.598666 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0927 14:57:16.598778 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":179}
I20260927 14:57:16.600618 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.568730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36483 (local address 127.0.0.1:42746)
0927 14:57:16.569012 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:16.569025 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:16.569659 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:16.570020 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:16.570023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:16.570276 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:16.570847 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.570858 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.571923 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.571927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.572801 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.572810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.573514 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.573517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.573608 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.595711 (+ 22103us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:16.595732 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:16.596436 (+ 704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:16.598416 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.598419 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.598421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.598666 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.599012 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.599014 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.599016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.599111 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.599970 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:16.599973 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:16.600067 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 14:57:16.600163 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 14:57:16.600316 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":20,"spinlock_wait_cycles":8448,"thread_start_us":90,"threads_started":1}
I20260927 14:57:16.601352 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.568423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42746 (local address 127.0.98.61:36483)
0927 14:57:16.568644 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:16.568649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:16.569584 (+ 935us) server_negotiation.cc:407] Connection header received
0927 14:57:16.569767 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:16.569771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:16.569821 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:16.569881 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:16.570991 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.571793 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.572932 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.573287 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.573358 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.596619 (+ 23261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:16.596624 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:16.596626 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:16.596662 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:16.598281 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.598778 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.598782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.598784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.598905 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.599580 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.599584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.599586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.599822 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:16.599928 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:16.601055 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0927 14:57:16.601198 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260927 14:57:16.604349 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.581818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41351 (local address 127.0.0.1:35624)
0927 14:57:16.581922 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:16.581936 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:16.582008 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:16.582403 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:16.582406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:16.582592 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:16.582681 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.582687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.583505 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.583508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.584057 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.584065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.584636 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.584640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.584740 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.585201 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:16.585215 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:16.586701 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:16.602498 (+ 15797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.602502 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.602504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.602934 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.603292 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.603295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.603297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.603400 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.603808 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:16.603811 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:16.603903 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 14:57:16.604006 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 14:57:16.604177 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 14:57:16.604369 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.581881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35624 (local address 127.0.98.60:41351)
0927 14:57:16.582172 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:16.582178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:16.582188 (+ 10us) server_negotiation.cc:407] Connection header received
0927 14:57:16.582240 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:16.582243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:16.582283 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:16.582357 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:16.582830 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.583400 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.584165 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.584518 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.584612 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.586914 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:16.586917 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:16.586920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:16.586946 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:16.602026 (+ 15080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.603066 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.603070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.603071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.603184 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.603505 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.603507 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.603508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.603702 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:16.603791 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:16.604079 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 14:57:16.604197 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":116,"threads_started":1}
I20260927 14:57:16.605379 6488 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:41351: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 41351
}
attrs {
promote: true
}
}
dest_uuid: "3e1e4b37da644604875d93c9bf1060ba"
cas_config_opid_index: 5
I20260927 14:57:16.605969 6488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: true } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: true } }
I20260927 14:57:16.607522 7049 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 56ffb5d4c52e4a5fa29276ae2214dc99 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: true } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: true } } }
I20260927 14:57:16.608826 7049 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:41351
I20260927 14:57:16.609022 7048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3e1e4b37da644604875d93c9bf1060ba" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: true } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: true } } }
I20260927 14:57:16.609155 7048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:16.610096 7182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:16.611605 7182 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [term 1 FOLLOWER]: Refusing update from remote peer 3e1e4b37da644604875d93c9bf1060ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 14:57:16.612229 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:57:16.613592 7061 master.cc:537] Master@127.0.98.60:41351 shutting down...
I20260927 14:57:16.617519 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.607179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36483 (local address 127.0.0.1:42750)
0927 14:57:16.609507 (+ 2328us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:16.609522 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:16.609615 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:16.610062 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:16.610065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:16.610084 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:16.610182 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.610189 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.611222 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.611225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.613416 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.613427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.615198 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.615203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.615661 (+ 458us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.615724 (+ 63us) client_negotiation.cc:769] Sending connection context
0927 14:57:16.615810 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 14:57:16.615908 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2240}
I20260927 14:57:16.618148 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.607428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42750 (local address 127.0.98.61:36483)
0927 14:57:16.607552 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:16.607557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:16.609704 (+ 2147us) server_negotiation.cc:407] Connection header received
0927 14:57:16.609763 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:16.609766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:16.609829 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:16.609926 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:16.610309 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.611091 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.613939 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.614834 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.614952 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.615124 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:16.617893 (+ 2769us) server_negotiation.cc:299] Negotiation successful
0927 14:57:16.618008 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 14:57:16.618659 7100 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [term 1 LEARNER]: Refusing update from remote peer 3e1e4b37da644604875d93c9bf1060ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 14:57:16.619208 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 14:57:16.619449 7182 raft_consensus.cc:858] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41351 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: true } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: true } }
W20260927 14:57:16.619678 7182 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41351 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: true } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: true } }
I20260927 14:57:16.621222 7100 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 56ffb5d4c52e4a5fa29276ae2214dc99 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: true } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: true } } }
I20260927 14:57:16.621537 7182 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 86c641e5060c4fe88d0b39cf0550cd6f (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: true } } }
I20260927 14:57:16.623766 7129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e1e4b37da644604875d93c9bf1060ba. Latest consensus state: current_term: 1 leader_uuid: "3e1e4b37da644604875d93c9bf1060ba" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: true } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: true } } }
I20260927 14:57:16.623926 7129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [sys.catalog]: This master's current role is: LEARNER
I20260927 14:57:16.623896 7214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e1e4b37da644604875d93c9bf1060ba. Latest consensus state: current_term: 1 leader_uuid: "3e1e4b37da644604875d93c9bf1060ba" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: true } } }
I20260927 14:57:16.623988 7214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 14:57:16.624022 7129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e1e4b37da644604875d93c9bf1060ba" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: true } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: true } } }
I20260927 14:57:16.624089 7129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [sys.catalog]: This master's current role is: LEARNER
I20260927 14:57:16.624066 7214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e1e4b37da644604875d93c9bf1060ba" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: true } } }
I20260927 14:57:16.624123 7214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 14:57:16.624183 7214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e1e4b37da644604875d93c9bf1060ba" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: true } } }
I20260927 14:57:16.624234 7214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 14:57:16.625697 7050 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba: attempting to promote NON_VOTER 86c641e5060c4fe88d0b39cf0550cd6f to VOTER
I20260927 14:57:16.626185 7050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: true } }
I20260927 14:57:16.628041 7100 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [term 1 LEARNER]: Refusing update from remote peer 3e1e4b37da644604875d93c9bf1060ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 14:57:16.628437 7049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260927 14:57:16.628564 6472 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba -> Peer 56ffb5d4c52e4a5fa29276ae2214dc99 (127.0.98.60:41351): Couldn't send request to peer 56ffb5d4c52e4a5fa29276ae2214dc99. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 14:57:16.629089 7061 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260927 14:57:16.629320 7061 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260927 14:57:16.629412 7061 tablet_replica.cc:330] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99: stopping tablet replica
I20260927 14:57:16.631008 7050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 86c641e5060c4fe88d0b39cf0550cd6f (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: true } } }
I20260927 14:57:16.631599 7100 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 86c641e5060c4fe88d0b39cf0550cd6f (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: true } } }
I20260927 14:57:16.632578 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3e1e4b37da644604875d93c9bf1060ba" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: true } } }
I20260927 14:57:16.632683 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:16.632823 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e1e4b37da644604875d93c9bf1060ba" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: true } } }
I20260927 14:57:16.632908 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:16.633008 7240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e1e4b37da644604875d93c9bf1060ba" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: true } } }
I20260927 14:57:16.633137 7240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:57:16.651952 7061 master.cc:552] Master@127.0.98.60:41351 shutdown complete.
I20260927 14:57:16.652016 7061 master_runner.cc:305] Clearing existing system tablet
I20260927 14:57:16.652230 7061 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:16.652276 7061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:16.653348 7061 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 14:57:16.654130 7246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:16.654359 7061 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:16.654491 7061 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 14:57:16.654526 7061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-2/wal
uuid: "56ffb5d4c52e4a5fa29276ae2214dc99"
format_stamp: "Formatted at 2026-09-27 14:57:16 on dist-test-slave-pz0h"
server_key: "45f7487fe34909a7bdb12d736aa610bc"
server_key_iv: "c6f475fdadc96eb47907c1a07cd00cbb"
server_key_version: "encryptionkey@0"
I20260927 14:57:16.656111 7061 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:57:16.658385 7061 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 14:57:16.658564 7061 log.cc:1196] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 14:57:16.658942 7061 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99: Deleting consensus metadata
I20260927 14:57:16.659101 7061 master_runner.cc:315] Copying system tablet from 127.0.98.62:44545
I20260927 14:57:16.660193 7061 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:44545
I20260927 14:57:16.668107 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.660817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56738 (local address 127.0.98.62:44545)
0927 14:57:16.660943 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:16.660948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:16.661120 (+ 172us) server_negotiation.cc:407] Connection header received
0927 14:57:16.661254 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:16.661258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:16.661303 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:16.661367 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:16.661790 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.662318 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.663614 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.663863 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.663951 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.665016 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:16.665019 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:16.665023 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:16.665050 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:16.666461 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.666883 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.666886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.666888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.666980 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.667272 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.667275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.667277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.667485 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:16.667571 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:16.667822 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 14:57:16.667933 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 14:57:16.668118 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.660737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.0.1:56738)
0927 14:57:16.661038 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:16.661049 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:16.661163 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:16.661392 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:16.661394 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:16.661586 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:16.661681 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.661686 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.662430 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.662433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.663498 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.663507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.663969 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.663973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.664065 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.664544 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:16.664556 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:16.664881 (+ 325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:16.666572 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.666576 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.666578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.666777 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.667082 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.667084 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.667086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.667188 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.667571 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:16.667574 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:16.667673 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 14:57:16.667759 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 14:57:16.667920 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":83,"threads_started":1}
I20260927 14:57:16.668891 6508 tablet_copy_service.cc:140] P 3e1e4b37da644604875d93c9bf1060ba: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 56ffb5d4c52e4a5fa29276ae2214dc99 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:56738)
I20260927 14:57:16.669014 6508 tablet_copy_service.cc:161] P 3e1e4b37da644604875d93c9bf1060ba: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 56ffb5d4c52e4a5fa29276ae2214dc99 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:56738: session id = 56ffb5d4c52e4a5fa29276ae2214dc99-00000000000000000000000000000000
I20260927 14:57:16.670107 6508 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 14:57:16.671382 7061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 14:57:16.673512 7061 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99: tablet copy: Starting download of 0 data blocks...
I20260927 14:57:16.673703 7061 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99: tablet copy: Starting download of 1 WAL segments...
I20260927 14:57:16.675136 7061 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 14:57:16.676767 6508 tablet_copy_service.cc:342] P 3e1e4b37da644604875d93c9bf1060ba: Request end of tablet copy session 56ffb5d4c52e4a5fa29276ae2214dc99-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:56738
I20260927 14:57:16.676852 6508 tablet_copy_service.cc:433] P 3e1e4b37da644604875d93c9bf1060ba: ending tablet copy session 56ffb5d4c52e4a5fa29276ae2214dc99-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 56ffb5d4c52e4a5fa29276ae2214dc99
I20260927 14:57:16.686093 7061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:16.687770 7253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:16.687788 7256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:16.688019 7061 server_base.cc:1034] running on GCE node
W20260927 14:57:16.688273 7254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:16.688915 7061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:16.689195 7061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:16.690325 7061 hybrid_clock.cc:648] HybridClock initialized: now 1790521036690311 us; error 30 us; skew 500 ppm
Sep 27 14:57:16 dist-test-slave-pz0h krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521036, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:16.691687 7061 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 14:57:16.692131 7061 webserver.cc:460] Webserver started at http://127.0.98.60:35931/ using document root <none> and password file <none>
I20260927 14:57:16.692309 7061 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:16.692343 7061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:16.693392 7061 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 14:57:16.694344 7263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:16.694561 7061 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 14:57:16.694669 7061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-2/wal
uuid: "56ffb5d4c52e4a5fa29276ae2214dc99"
format_stamp: "Formatted at 2026-09-27 14:57:16 on dist-test-slave-pz0h"
server_key: "45f7487fe34909a7bdb12d736aa610bc"
server_key_iv: "c6f475fdadc96eb47907c1a07cd00cbb"
server_key_version: "encryptionkey@0"
I20260927 14:57:16.694757 7061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:16.699553 6813 heartbeater.cc:503] Master 127.0.98.62:44545 was elected leader, sending a full tablet report...
I20260927 14:57:16.702354 7061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:16.704421 7061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:16.708853 7061 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41351
I20260927 14:57:16.708863 7315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41351 every 8 connection(s)
I20260927 14:57:16.709379 7061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/master-2/data/info.pb
I20260927 14:57:16.710856 7316 sys_catalog.cc:263] Verifying existing consensus state
I20260927 14:57:16.711711 7316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99: Bootstrap starting.
I20260927 14:57:16.719143 7316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 14:57:16.719429 7316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99: Bootstrap complete.
I20260927 14:57:16.719960 7316 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: true } }
I20260927 14:57:16.720081 7316 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 56ffb5d4c52e4a5fa29276ae2214dc99, State: Initialized, Role: LEARNER
I20260927 14:57:16.720183 7316 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: true } }
I20260927 14:57:16.720643 7316 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:57:16.720678 7319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: true } } }
I20260927 14:57:16.720896 7319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [sys.catalog]: This master's current role is: LEARNER
I20260927 14:57:16.724634 7330 catalog_manager.cc:1259] Loaded cluster ID: 2dae8741e60142f9ba09d6c7b3a0d359
I20260927 14:57:16.724704 7330 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99: loading cluster ID for follower catalog manager: success
I20260927 14:57:16.726382 7330 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99: acquiring CA information for follower catalog manager: success
I20260927 14:57:16.727772 7330 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 14:57:16.755831 7235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:16.756256 7235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:16.756320 7235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:16.762739 7235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:16.762879 7235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:16.762980 7235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:57:16.770982 7235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40249
--tserver_master_addrs=127.0.98.62:44545,127.0.98.61:36483,127.0.98.60:41351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:16.772405 7235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:16.773640 7235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:16.783788 7335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:16.784081 7235 server_base.cc:1034] running on GCE node
W20260927 14:57:16.783790 7338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:16.783790 7336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:16.784806 7235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:16.785562 7235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:16.786738 7235 hybrid_clock.cc:648] HybridClock initialized: now 1790521036786716 us; error 41 us; skew 500 ppm
Sep 27 14:57:16 dist-test-slave-pz0h krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521036, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:16.789633 7235 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 14:57:16.790637 7235 webserver.cc:460] Webserver started at http://127.0.98.1:40249/ using document root <none> and password file <none>
I20260927 14:57:16.791260 7235 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:16.791340 7235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:16.795686 7235 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 14:57:16.798002 7345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:16.799383 7235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:57:16.799501 7235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "1544dca4984d47c494bb602d886ce11e"
format_stamp: "Formatted at 2026-09-27 14:57:15 on dist-test-slave-pz0h"
server_key: "126938ac4ca59442bc4c63192ad7495e"
server_key_iv: "98bd2fdd956366e361ffd9405f379077"
server_key_version: "encryptionkey@0"
I20260927 14:57:16.800096 7235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:16.827175 7235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:16.830905 7235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:16.831121 7235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:16.832237 7235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:16.833243 7235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:16.833290 7235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:16.833333 7235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:16.833361 7235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:16.842738 7235 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34875
I20260927 14:57:16.842844 7458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34875 every 8 connection(s)
I20260927 14:57:16.843976 7235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:57:16.846256 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 7235
I20260927 14:57:16.846436 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 6684
I20260927 14:57:16.853307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35207
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44545
--webserver_port=38005
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:44545,127.0.98.61:36483
--tserver_master_addrs=127.0.98.62:44545,127.0.98.61:36483,127.0.98.60:41351 with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 14:57:16 dist-test-slave-pz0h krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521036, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 14:57:16 dist-test-slave-pz0h krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521036, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 14:57:16 dist-test-slave-pz0h krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521036, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 14:57:16.861841 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.847157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34337 (local address 127.0.98.60:41351)
0927 14:57:16.849067 (+ 1910us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:16.849071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:16.849086 (+ 15us) server_negotiation.cc:407] Connection header received
0927 14:57:16.849127 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:16.849131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:16.849174 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:16.849261 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:16.850875 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.852081 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.853290 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.853654 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.853720 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.857440 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:16.857445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:16.857447 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:16.857478 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:16.860025 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.860521 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.860524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.860526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.860607 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.860891 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.860895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.860896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.861115 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:16.861193 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:16.861528 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 14:57:16.861665 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1835,"thread_start_us":84,"threads_started":1}
I20260927 14:57:16.861836 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.846062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36019 (local address 127.0.98.62:44545)
0927 14:57:16.846287 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:16.846290 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:16.847844 (+ 1554us) server_negotiation.cc:407] Connection header received
0927 14:57:16.848834 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:16.848838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:16.848896 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:16.848941 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:16.850715 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.851523 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.853289 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.853645 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.853721 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.857547 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:16.857552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:16.857554 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:16.857583 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:16.859337 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.859951 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.859955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.859957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.860046 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.860405 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.860409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.860410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.860658 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:16.860727 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:16.861543 (+ 816us) server_negotiation.cc:299] Negotiation successful
0927 14:57:16.861662 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260927 14:57:16.861850 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.845930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53063 (local address 127.0.98.61:36483)
0927 14:57:16.846202 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:16.846206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:16.849558 (+ 3352us) server_negotiation.cc:407] Connection header received
0927 14:57:16.849712 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:16.849715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:16.849766 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:16.849814 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:16.851345 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.852088 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.853280 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.853580 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.853655 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.857440 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:16.857445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:16.857448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:16.857481 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:16.859108 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.859856 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.859861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.859863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.859972 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:16.860401 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:16.860405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:16.860407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:16.860658 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:16.860747 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:16.861553 (+ 806us) server_negotiation.cc:299] Negotiation successful
0927 14:57:16.861691 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260927 14:57:16.863750 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.846374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36483 (local address 127.0.98.1:53063)
0927 14:57:16.849477 (+ 3103us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:16.849490 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:16.849618 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:16.849952 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:16.849955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:16.850133 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:16.850723 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.850731 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.852232 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.852235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.853148 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.853158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.853863 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.853867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.854097 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.854851 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:16.854884 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:16.857232 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:16.859269 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.859276 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.859298 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.859724 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.860089 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.860093 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.860095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.860291 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.860806 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:16.860809 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:16.861244 (+ 435us) client_negotiation.cc:769] Sending connection context
0927 14:57:16.862518 (+ 1274us) client_negotiation.cc:240] Negotiation successful
0927 14:57:16.862678 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3015,"mutex_wait_us":1273,"spinlock_wait_cycles":30208,"thread_start_us":77,"threads_started":1}
I20260927 14:57:16.863773 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.846374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.98.1:36019)
0927 14:57:16.847708 (+ 1334us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:16.847750 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:16.848593 (+ 843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:16.849207 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:16.849214 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:16.849800 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:16.850543 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.850562 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.851683 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.851687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.853157 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.853167 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.853874 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.853878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.854110 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.854889 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:16.854904 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:16.857392 (+ 2488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:16.859563 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.859566 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.859568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.859822 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.860154 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.860157 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.860160 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.860296 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.860799 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:16.860807 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:16.861243 (+ 436us) client_negotiation.cc:769] Sending connection context
0927 14:57:16.862063 (+ 820us) client_negotiation.cc:240] Negotiation successful
0927 14:57:16.862575 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1130,"mutex_wait_us":335,"spinlock_wait_cycles":4480,"thread_start_us":123,"threads_started":1}
I20260927 14:57:16.863997 7463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:16.847112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41351 (local address 127.0.98.1:34337)
0927 14:57:16.848333 (+ 1221us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:16.848346 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:16.848604 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:16.849288 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:16.849291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:16.849805 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:16.850543 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.850562 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.852214 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.852217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.853157 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:16.853167 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:16.853874 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:16.853878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:16.854135 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:16.854981 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:16.855006 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:16.857207 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:16.860154 (+ 2947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.860157 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.860160 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.860416 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.860711 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:16.860714 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:16.860716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:16.860796 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:16.861247 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:16.861249 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:16.861335 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 14:57:16.862063 (+ 728us) client_negotiation.cc:240] Negotiation successful
0927 14:57:16.862513 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1159,"mutex_wait_us":278,"spinlock_wait_cycles":10112,"thread_start_us":131,"threads_started":1}
I20260927 14:57:16.865579 7461 heartbeater.cc:347] Connected to a master server at 127.0.98.61:36483
I20260927 14:57:16.865676 7459 heartbeater.cc:347] Connected to a master server at 127.0.98.60:41351
I20260927 14:57:16.865823 7461 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:16.865823 7459 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:16.865849 7460 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44545
I20260927 14:57:16.865949 7460 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:16.866524 7461 heartbeater.cc:511] Master 127.0.98.61:36483 requested a full tablet report, sending...
I20260927 14:57:16.866524 7460 heartbeater.cc:511] Master 127.0.98.62:44545 requested a full tablet report, sending...
I20260927 14:57:16.866524 7459 heartbeater.cc:511] Master 127.0.98.60:41351 requested a full tablet report, sending...
I20260927 14:57:16.867908 6488 ts_manager.cc:194] Re-registered known tserver with Master: 1544dca4984d47c494bb602d886ce11e (127.0.98.1:34875)
I20260927 14:57:16.868216 7276 ts_manager.cc:194] Registered new tserver with Master: 1544dca4984d47c494bb602d886ce11e (127.0.98.1:34875)
I20260927 14:57:16.868227 7082 ts_manager.cc:194] Registered new tserver with Master: 1544dca4984d47c494bb602d886ce11e (127.0.98.1:34875)
I20260927 14:57:16.868690 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36019
I20260927 14:57:16.949823 6947 heartbeater.cc:503] Master 127.0.98.62:44545 was elected leader, sending a full tablet report...
W20260927 14:57:16.970602 7467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:16.970944 7467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:16.970999 7467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:16.975224 7467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:16.975308 7467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:16.975369 7467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:57:16.981024 7467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38005
--tserver_master_addrs=127.0.98.62:44545,127.0.98.61:36483,127.0.98.60:41351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:16.982144 7467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:16.983340 7467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:16.993310 7472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:16.993310 7473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:16.993376 7467 server_base.cc:1034] running on GCE node
W20260927 14:57:16.993312 7475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:16.993902 7467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:16.994616 7467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:16.995802 7467 hybrid_clock.cc:648] HybridClock initialized: now 1790521036995760 us; error 49 us; skew 500 ppm
Sep 27 14:57:16 dist-test-slave-pz0h krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521036, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:16.998602 7467 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 14:57:16.999536 7467 webserver.cc:460] Webserver started at http://127.0.98.2:38005/ using document root <none> and password file <none>
I20260927 14:57:17.000084 7467 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:17.000157 7467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:17.004130 7467 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:57:17.006232 7482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:17.007522 7467 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:57:17.007637 7467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "5e76f2b076e14fcca79b8699d6be4704"
format_stamp: "Formatted at 2026-09-27 14:57:15 on dist-test-slave-pz0h"
server_key: "bb97c4f6727f7a81d7dbfed928679e29"
server_key_iv: "a3ef76829e051e87261505c238f58c4f"
server_key_version: "encryptionkey@0"
I20260927 14:57:17.008190 7467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:17.038851 7467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:17.041802 7467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:17.041980 7467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:17.042577 7467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:17.043463 7467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:17.043510 7467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:17.043565 7467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:17.043592 7467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:17.052277 7467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35207
I20260927 14:57:17.052297 7595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35207 every 8 connection(s)
I20260927 14:57:17.053293 7467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:57:17.060912 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 7467
I20260927 14:57:17.061086 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 6818
Sep 27 14:57:17 dist-test-slave-pz0h krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521036, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 14:57:17 dist-test-slave-pz0h krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521036, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 14:57:17.068347 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42813
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44545
--webserver_port=33941
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:44545,127.0.98.61:36483
--tserver_master_addrs=127.0.98.62:44545,127.0.98.61:36483,127.0.98.60:41351 with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:57:17.069779 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.055231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53335 (local address 127.0.98.61:36483)
0927 14:57:17.055447 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:17.055451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:17.056783 (+ 1332us) server_negotiation.cc:407] Connection header received
0927 14:57:17.057411 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:17.057415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:17.057469 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:17.057520 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:17.059780 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.060624 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.061783 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.062091 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.062163 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.065435 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:17.065440 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:17.065443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:17.065476 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:17.067186 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:17.067855 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:17.067858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:17.067860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:17.067967 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:17.068397 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:17.068400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:17.068402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:17.068642 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:17.068712 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:17.069471 (+ 759us) server_negotiation.cc:299] Negotiation successful
0927 14:57:17.069611 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260927 14:57:17.069855 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.055213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33645 (local address 127.0.98.62:44545)
0927 14:57:17.055360 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:17.055365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:17.056871 (+ 1506us) server_negotiation.cc:407] Connection header received
0927 14:57:17.057385 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:17.057388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:17.057465 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:17.057540 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:17.059117 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.059649 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.061406 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.061980 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.062064 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.065617 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:17.065621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:17.065623 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:17.065653 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:17.067153 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:17.067854 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:17.067858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:17.067860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:17.067961 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:17.068409 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:17.068413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:17.068415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:17.068639 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:17.068712 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:17.069545 (+ 833us) server_negotiation.cc:299] Negotiation successful
0927 14:57:17.069682 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 14:57:17 dist-test-slave-pz0h krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521036, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 14:57:17.071938 7602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.055619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36483 (local address 127.0.98.2:53335)
0927 14:57:17.056681 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:17.056694 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:17.057190 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:17.057740 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:17.057748 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:17.058308 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:17.058938 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:17.058958 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.060768 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.060772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:17.061639 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:17.061650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.062319 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.062322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:17.062470 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.063413 (+ 943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:17.063428 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:17.065197 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:17.067328 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:17.067331 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:17.067334 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:17.067744 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:17.068073 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:17.068076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:17.068078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:17.068286 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:17.069027 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:17.069030 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:17.069192 (+ 162us) client_negotiation.cc:769] Sending connection context
0927 14:57:17.070323 (+ 1131us) client_negotiation.cc:240] Negotiation successful
0927 14:57:17.070693 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":983,"mutex_wait_us":174,"spinlock_wait_cycles":158208,"thread_start_us":143,"threads_started":1}
I20260927 14:57:17.072211 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.055619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.98.2:33645)
0927 14:57:17.056786 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:17.056799 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:17.057193 (+ 394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:17.057739 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:17.057747 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:17.058308 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:17.058943 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:17.058961 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.059792 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.059795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:17.061265 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:17.061276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.062269 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.062271 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:17.062462 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.063316 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:17.063338 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:17.065464 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:17.067293 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:17.067300 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:17.067321 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:17.067740 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:17.068080 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:17.068082 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:17.068084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:17.068296 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:17.068797 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:17.068805 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:17.069363 (+ 558us) client_negotiation.cc:769] Sending connection context
0927 14:57:17.070224 (+ 861us) client_negotiation.cc:240] Negotiation successful
0927 14:57:17.070776 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1082,"mutex_wait_us":306,"spinlock_wait_cycles":25088,"thread_start_us":122,"threads_started":1}
I20260927 14:57:17.073551 7597 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44545
I20260927 14:57:17.073822 7597 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:17.074456 7597 heartbeater.cc:511] Master 127.0.98.62:44545 requested a full tablet report, sending...
I20260927 14:57:17.073547 7598 heartbeater.cc:347] Connected to a master server at 127.0.98.61:36483
I20260927 14:57:17.074942 7598 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:17.075109 7598 heartbeater.cc:511] Master 127.0.98.61:36483 requested a full tablet report, sending...
I20260927 14:57:17.075690 6488 ts_manager.cc:194] Re-registered known tserver with Master: 5e76f2b076e14fcca79b8699d6be4704 (127.0.98.2:35207)
I20260927 14:57:17.075835 7082 ts_manager.cc:194] Registered new tserver with Master: 5e76f2b076e14fcca79b8699d6be4704 (127.0.98.2:35207)
I20260927 14:57:17.075914 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.055348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46259 (local address 127.0.98.60:41351)
0927 14:57:17.055457 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:17.055460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:17.056373 (+ 913us) server_negotiation.cc:407] Connection header received
0927 14:57:17.057411 (+ 1038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:17.057415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:17.057470 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:17.057520 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:17.060772 (+ 3252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.061503 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.068963 (+ 7460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.069311 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.069396 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.072185 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:17.072189 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:17.072192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:17.072224 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:17.073919 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:17.074338 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:17.074342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:17.074344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:17.074417 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:17.074687 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:17.074690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:17.074692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:17.074941 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:17.075033 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:17.075636 (+ 603us) server_negotiation.cc:299] Negotiation successful
0927 14:57:17.075767 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 14:57:17.076368 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.055619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41351 (local address 127.0.98.2:46259)
0927 14:57:17.056204 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:17.056264 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:17.057189 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:17.057740 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:17.057748 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:17.058308 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:17.058938 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:17.058958 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.067956 (+ 8998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.067960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:17.068821 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:17.068832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.069582 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.069586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:17.069678 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.070216 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:17.070229 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:17.071988 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:17.074046 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:17.074048 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:17.074050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:17.074230 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:17.074502 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:17.074505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:17.074507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:17.074592 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:17.075349 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:17.075352 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:17.075454 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 14:57:17.076041 (+ 587us) client_negotiation.cc:240] Negotiation successful
0927 14:57:17.076195 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":50,"spinlock_wait_cycles":12032,"thread_start_us":601,"threads_started":1}
I20260927 14:57:17.076473 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33645
I20260927 14:57:17.076952 7596 heartbeater.cc:347] Connected to a master server at 127.0.98.60:41351
I20260927 14:57:17.077035 7596 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:17.077186 7596 heartbeater.cc:511] Master 127.0.98.60:41351 requested a full tablet report, sending...
I20260927 14:57:17.077853 7276 ts_manager.cc:194] Registered new tserver with Master: 5e76f2b076e14fcca79b8699d6be4704 (127.0.98.2:35207)
I20260927 14:57:17.173856 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.168975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35640 (local address 127.0.98.60:41351)
0927 14:57:17.169120 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:17.169124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:17.169289 (+ 165us) server_negotiation.cc:407] Connection header received
0927 14:57:17.169434 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:17.169437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:17.169504 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:17.169615 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:17.169886 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.170704 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.172219 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.172929 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.173045 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.173218 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:17.173598 (+ 380us) server_negotiation.cc:299] Negotiation successful
0927 14:57:17.173694 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 14:57:17.174466 7604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.168840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41351 (local address 127.0.0.1:35640)
0927 14:57:17.169190 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:17.169204 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:17.169327 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:17.169636 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:17.169638 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:17.169660 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:17.169770 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:17.169776 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.170859 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.170862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:17.172073 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:17.172085 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.173289 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.173293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:17.173403 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.173473 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 14:57:17.174200 (+ 727us) client_negotiation.cc:240] Negotiation successful
0927 14:57:17.174304 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":92,"threads_started":1}
I20260927 14:57:17.175176 7289 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 14:57:17.175457 7319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e1e4b37da644604875d93c9bf1060ba. Latest consensus state: current_term: 1 leader_uuid: "3e1e4b37da644604875d93c9bf1060ba" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: true } } }
I20260927 14:57:17.175603 7319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [sys.catalog]: This master's current role is: LEARNER
W20260927 14:57:17.188999 7603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:17.189321 7603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:17.189385 7603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:17.193807 7603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:17.193890 7603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:17.193974 7603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:57:17.199878 7603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33941
--tserver_master_addrs=127.0.98.62:44545,127.0.98.61:36483,127.0.98.60:41351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:17.201013 7603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:17.202049 7603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:17.211910 7612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:17.211987 7610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:17.211892 7609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:17.212594 7603 server_base.cc:1034] running on GCE node
I20260927 14:57:17.213009 7603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:17.213697 7603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:17.214871 7603 hybrid_clock.cc:648] HybridClock initialized: now 1790521037214840 us; error 48 us; skew 500 ppm
Sep 27 14:57:17 dist-test-slave-pz0h krb5kdc[6443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521037, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:17.217939 7603 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 14:57:17.219183 7603 webserver.cc:460] Webserver started at http://127.0.98.3:33941/ using document root <none> and password file <none>
I20260927 14:57:17.219913 7603 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:17.220019 7603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:17.225481 7603 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:57:17.228367 7619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:17.229995 7603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 14:57:17.230129 7603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "280b4c9e2726475989c4e06525b89aec"
format_stamp: "Formatted at 2026-09-27 14:57:15 on dist-test-slave-pz0h"
server_key: "7df4b743ca0ca3a4bfb230eeb6b4c0a3"
server_key_iv: "27ba8019dd30abbfce5c09cc4c53f6ac"
server_key_version: "encryptionkey@0"
I20260927 14:57:17.230954 7603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:17.249135 7603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:17.253455 7603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:17.253684 7603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:17.254442 7603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:17.255646 7603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:17.255718 7603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:17.255774 7603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:17.255805 7603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:17.266969 7603 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42813
I20260927 14:57:17.267004 7732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42813 every 8 connection(s)
I20260927 14:57:17.268323 7603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:57:17.276710 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 7603
Sep 27 14:57:17 dist-test-slave-pz0h krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521037, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 14:57:17 dist-test-slave-pz0h krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521037, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 14:57:17.287158 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.270245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48079 (local address 127.0.98.62:44545)
0927 14:57:17.270373 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:17.270378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:17.271517 (+ 1139us) server_negotiation.cc:407] Connection header received
0927 14:57:17.272791 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:17.272795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:17.272855 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:17.272908 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:17.275047 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.275895 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.278040 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.278404 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.278490 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.282892 (+ 4402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:17.282896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:17.282899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:17.282930 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:17.284546 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:17.285243 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:17.285247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:17.285249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:17.285346 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:17.285775 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:17.285779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:17.285781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:17.286001 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:17.286083 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:17.286841 (+ 758us) server_negotiation.cc:299] Negotiation successful
0927 14:57:17.286973 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 14:57:17.288916 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.270706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.98.3:48079)
0927 14:57:17.271337 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:17.271395 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:17.272543 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:17.273330 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:17.273338 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:17.274023 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:17.274879 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:17.274890 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.276088 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.276091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:17.277848 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:17.277860 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.278649 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.278652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:17.278754 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.279489 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:17.279511 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:17.282699 (+ 3188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:17.284679 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:17.284688 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:17.284712 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:17.285119 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:17.285464 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:17.285466 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:17.285468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:17.285671 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:17.286150 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:17.286158 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:17.286530 (+ 372us) client_negotiation.cc:769] Sending connection context
0927 14:57:17.287337 (+ 807us) client_negotiation.cc:240] Negotiation successful
0927 14:57:17.287754 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":716,"spinlock_wait_cycles":20992,"thread_start_us":640,"threads_started":1}
I20260927 14:57:17.291900 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.271827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34863 (local address 127.0.98.60:41351)
0927 14:57:17.271948 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:17.271953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:17.272080 (+ 127us) server_negotiation.cc:407] Connection header received
0927 14:57:17.272783 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:17.272787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:17.272965 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:17.273056 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:17.275043 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.275893 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.277887 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.278193 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.278275 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.282402 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:17.282406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:17.282409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:17.282439 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:17.285643 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:17.286734 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:17.286737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:17.286739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:17.286848 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:17.287144 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:17.287148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:17.287149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:17.287348 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:17.287424 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:17.288560 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0927 14:57:17.288954 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 14:57:17.292997 7734 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44545
I20260927 14:57:17.293306 7734 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:17.293862 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.270707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41351 (local address 127.0.98.3:34863)
0927 14:57:17.271999 (+ 1292us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:17.272012 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:17.272544 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:17.273328 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:17.273338 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:17.274021 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:17.274868 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:17.274890 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.276087 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.276091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:17.277734 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:17.277746 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.278505 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.278509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:17.278725 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.279572 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:17.279586 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:17.282180 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:17.285782 (+ 3602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:17.285785 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:17.285787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:17.286029 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:17.286954 (+ 925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:17.286956 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:17.286958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:17.287044 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:17.287737 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:17.287740 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:17.287834 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 14:57:17.293459 (+ 5625us) client_negotiation.cc:240] Negotiation successful
0927 14:57:17.293652 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1212,"mutex_wait_us":275,"spinlock_wait_cycles":75776,"thread_start_us":168,"threads_started":1}
I20260927 14:57:17.294564 7734 heartbeater.cc:511] Master 127.0.98.62:44545 requested a full tablet report, sending...
I20260927 14:57:17.295729 7733 heartbeater.cc:347] Connected to a master server at 127.0.98.60:41351
I20260927 14:57:17.295820 7733 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:17.296010 7733 heartbeater.cc:511] Master 127.0.98.60:41351 requested a full tablet report, sending...
Sep 27 14:57:17 dist-test-slave-pz0h krb5kdc[6443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521037, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 14:57:17.296554 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.283811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35644 (local address 127.0.98.60:41351)
0927 14:57:17.284431 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:17.284435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:17.284740 (+ 305us) server_negotiation.cc:407] Connection header received
0927 14:57:17.284784 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:17.284786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:17.284837 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:17.284917 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:17.285787 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.286590 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.287779 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.288104 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.288188 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.289842 (+ 1654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:17.289846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:17.289848 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:17.289873 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:17.292897 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:17.294760 (+ 1863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:17.294766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:17.294768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:17.294902 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:17.295298 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:17.295302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:17.295304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:17.295546 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:17.295672 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:17.296221 (+ 549us) server_negotiation.cc:299] Negotiation successful
0927 14:57:17.296363 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":99,"threads_started":1}
I20260927 14:57:17.298115 7276 ts_manager.cc:194] Registered new tserver with Master: 280b4c9e2726475989c4e06525b89aec (127.0.98.3:42813)
I20260927 14:57:17.299454 6488 ts_manager.cc:194] Re-registered known tserver with Master: 280b4c9e2726475989c4e06525b89aec (127.0.98.3:42813)
I20260927 14:57:17.300223 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48079
I20260927 14:57:17.302002 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.272861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53443 (local address 127.0.98.61:36483)
0927 14:57:17.272976 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:17.272979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:17.273234 (+ 255us) server_negotiation.cc:407] Connection header received
0927 14:57:17.273382 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:17.273385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:17.273430 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:17.273480 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:17.275042 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.275893 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.287980 (+ 12087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.288300 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.288376 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.296930 (+ 8554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:17.296934 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:17.296937 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:17.296966 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:17.298541 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:17.300633 (+ 2092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:17.300637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:17.300639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:17.300741 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:17.301040 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:17.301043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:17.301045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:17.301269 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:17.301353 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:17.301722 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 14:57:17.301850 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 14:57:17.302573 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.272518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36483 (local address 127.0.98.3:53443)
0927 14:57:17.273164 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:17.273176 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:17.273289 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:17.273609 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:17.273611 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:17.274021 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:17.274868 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:17.274890 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.276087 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.276091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:17.286591 (+ 10500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:17.286610 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.293133 (+ 6523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.293137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:17.293266 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.294421 (+ 1155us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:17.294435 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:17.296758 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:17.298665 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:17.298669 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:17.298672 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:17.298967 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:17.300823 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:17.300826 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:17.300828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:17.300929 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:17.301434 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:17.301437 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:17.301533 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 14:57:17.302205 (+ 672us) client_negotiation.cc:240] Negotiation successful
0927 14:57:17.302369 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":259,"spinlock_wait_cycles":6144,"thread_start_us":93,"threads_started":1}
I20260927 14:57:17.303102 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.278824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56750 (local address 127.0.98.62:44545)
0927 14:57:17.280294 (+ 1470us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:17.280298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:17.280312 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:57:17.280350 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:17.280352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:17.280403 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:17.280494 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:17.281407 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.282264 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.284481 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.289700 (+ 5219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.289830 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.293241 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:17.293246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:17.293249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:17.293282 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:17.300348 (+ 7066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:17.300883 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:17.300888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:17.300890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:17.300984 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:17.301266 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:17.301270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:17.301272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:17.301516 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:17.301653 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:17.302749 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0927 14:57:17.302919 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1389,"thread_start_us":84,"threads_started":1}
I20260927 14:57:17.303161 7735 heartbeater.cc:347] Connected to a master server at 127.0.98.61:36483
I20260927 14:57:17.303249 7735 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:17.303668 7735 heartbeater.cc:511] Master 127.0.98.61:36483 requested a full tablet report, sending...
I20260927 14:57:17.304512 7082 ts_manager.cc:194] Registered new tserver with Master: 280b4c9e2726475989c4e06525b89aec (127.0.98.3:42813)
I20260927 14:57:17.306409 6488 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:57:17.309652 6488 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:57:17.310920 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.279634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42764 (local address 127.0.98.61:36483)
0927 14:57:17.280703 (+ 1069us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:17.280707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:17.281081 (+ 374us) server_negotiation.cc:407] Connection header received
0927 14:57:17.281135 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:17.281139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:17.281190 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:17.281280 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:17.282168 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.282971 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.291240 (+ 8269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.291650 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.291741 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.307021 (+ 15280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:17.307026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:17.307031 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:17.307070 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:17.308947 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:17.309505 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:17.309509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:17.309511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:17.309611 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:17.309918 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:17.309921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:17.309923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:17.310176 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:17.310311 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:17.310548 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 14:57:17.310692 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":992,"thread_start_us":84,"threads_started":1}
I20260927 14:57:17.313010 7049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e1e4b37da644604875d93c9bf1060ba" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: true } } }
I20260927 14:57:17.313159 7049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:17.315093 7049 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba: attempting to promote NON_VOTER 56ffb5d4c52e4a5fa29276ae2214dc99 to VOTER
I20260927 14:57:17.315433 7049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: false } }
I20260927 14:57:17.316542 7100 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [term 1 FOLLOWER]: Refusing update from remote peer 3e1e4b37da644604875d93c9bf1060ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 14:57:17.316543 7289 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [term 1 LEARNER]: Refusing update from remote peer 3e1e4b37da644604875d93c9bf1060ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 14:57:17.316876 7751 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 14:57:17.317018 7752 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 14:57:17.318943 7752 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 56ffb5d4c52e4a5fa29276ae2214dc99 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: false } } }
I20260927 14:57:17.319278 7289 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 56ffb5d4c52e4a5fa29276ae2214dc99 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: false } } }
I20260927 14:57:17.320706 7752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3e1e4b37da644604875d93c9bf1060ba" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: false } } }
I20260927 14:57:17.320760 7319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e1e4b37da644604875d93c9bf1060ba" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: false } } }
I20260927 14:57:17.320874 7319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:57:17.320940 7752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:17.321095 7752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e1e4b37da644604875d93c9bf1060ba" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: false } } }
I20260927 14:57:17.321218 7752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:17.320915 7751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e1e4b37da644604875d93c9bf1060ba" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: false } } }
I20260927 14:57:17.321522 7751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:17.321590 7100 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 56ffb5d4c52e4a5fa29276ae2214dc99 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: false } } }
I20260927 14:57:17.322973 7762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e1e4b37da644604875d93c9bf1060ba" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: false } } }
I20260927 14:57:17.323170 7762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:57:17.339310 7604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.332368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42813 (local address 127.0.0.1:38160)
0927 14:57:17.332475 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:17.332490 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:17.332569 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:17.334146 (+ 1577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:17.334149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:17.334167 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:17.334274 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:17.334281 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.335772 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.335775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:17.337244 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:17.337254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.338767 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.338771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:17.338915 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.338946 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 14:57:17.339038 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 14:57:17.339136 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":3}
I20260927 14:57:17.339332 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.332528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38160 (local address 127.0.98.3:42813)
0927 14:57:17.333532 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:17.333549 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:17.333582 (+ 33us) server_negotiation.cc:407] Connection header received
0927 14:57:17.333688 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:17.333695 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:17.333985 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:17.334431 (+ 446us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:17.334575 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.335666 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.337724 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.338616 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.338736 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.338949 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:17.339072 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 14:57:17.339183 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":874,"thread_start_us":120,"threads_started":1}
I20260927 14:57:17.342109 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.332579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35207 (local address 127.0.0.1:34210)
0927 14:57:17.333053 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:17.333066 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:17.333153 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:17.335450 (+ 2297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:17.335453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:17.335481 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:17.335584 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:17.335590 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.337323 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.337326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:17.338486 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:17.338497 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.341675 (+ 3178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.341679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:17.341787 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.341815 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 14:57:17.341862 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 14:57:17.341947 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":1,"thread_start_us":106,"threads_started":1}
I20260927 14:57:17.342921 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.332963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34210 (local address 127.0.98.2:35207)
0927 14:57:17.334189 (+ 1226us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:17.334200 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:17.334218 (+ 18us) server_negotiation.cc:407] Connection header received
0927 14:57:17.335104 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:17.335111 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:17.335304 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:17.335710 (+ 406us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:17.335937 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.337199 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.340658 (+ 3459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.341515 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.342429 (+ 914us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.342600 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:17.342654 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 14:57:17.342753 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1131,"thread_start_us":80,"threads_started":1}
I20260927 14:57:17.342891 7667 tablet_service.cc:1467] Processing CreateTablet for tablet eb37150bd2ec4141b6da2e66c2a5a6d1 (DEFAULT_TABLE table=test-table [id=ee9e448efec542e0bab734bd55496228]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:57:17.345374 7667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb37150bd2ec4141b6da2e66c2a5a6d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:17.346177 7530 tablet_service.cc:1467] Processing CreateTablet for tablet eb37150bd2ec4141b6da2e66c2a5a6d1 (DEFAULT_TABLE table=test-table [id=ee9e448efec542e0bab734bd55496228]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:57:17.347321 7530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb37150bd2ec4141b6da2e66c2a5a6d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:17.357010 7769 tablet_bootstrap.cc:492] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec: Bootstrap starting.
I20260927 14:57:17.357585 7768 tablet_bootstrap.cc:492] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 5e76f2b076e14fcca79b8699d6be4704: Bootstrap starting.
I20260927 14:57:17.357851 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.332813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34875 (local address 127.0.0.1:56296)
0927 14:57:17.333155 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:17.333168 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:17.333258 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:17.334564 (+ 1306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:17.334568 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:17.334587 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:17.334691 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:17.334696 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.336285 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.336289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:17.340456 (+ 4167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:17.340477 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.357313 (+ 16836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.357319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:17.357465 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.357499 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 14:57:17.357555 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 14:57:17.357646 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":2879,"thread_start_us":92,"threads_started":1}
I20260927 14:57:17.358614 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.333267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56296 (local address 127.0.98.1:34875)
0927 14:57:17.334084 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:17.334099 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:17.334123 (+ 24us) server_negotiation.cc:407] Connection header received
0927 14:57:17.334200 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:17.334205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:17.334396 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:17.334850 (+ 454us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:17.334987 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.336145 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.340631 (+ 4486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.341582 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.358036 (+ 16454us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.358239 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:17.358305 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 14:57:17.358421 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":609,"threads_started":1}
I20260927 14:57:17.359856 7769 tablet_bootstrap.cc:654] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:17.360781 7768 tablet_bootstrap.cc:654] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 5e76f2b076e14fcca79b8699d6be4704: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:17.360795 7769 log.cc:824] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec: Log is configured to *not* fsync() on all Append() calls
I20260927 14:57:17.361582 7768 log.cc:824] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 5e76f2b076e14fcca79b8699d6be4704: Log is configured to *not* fsync() on all Append() calls
I20260927 14:57:17.361558 7393 tablet_service.cc:1467] Processing CreateTablet for tablet eb37150bd2ec4141b6da2e66c2a5a6d1 (DEFAULT_TABLE table=test-table [id=ee9e448efec542e0bab734bd55496228]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:57:17.362848 7393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb37150bd2ec4141b6da2e66c2a5a6d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:17.363410 7769 tablet_bootstrap.cc:492] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec: No bootstrap required, opened a new log
I20260927 14:57:17.363632 7769 ts_tablet_manager.cc:1397] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 14:57:17.363740 7768 tablet_bootstrap.cc:492] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 5e76f2b076e14fcca79b8699d6be4704: No bootstrap required, opened a new log
I20260927 14:57:17.363940 7768 ts_tablet_manager.cc:1397] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 5e76f2b076e14fcca79b8699d6be4704: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 14:57:17.368252 7768 raft_consensus.cc:348] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 5e76f2b076e14fcca79b8699d6be4704 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "280b4c9e2726475989c4e06525b89aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 } } peers { permanent_uuid: "5e76f2b076e14fcca79b8699d6be4704" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35207 } } peers { permanent_uuid: "1544dca4984d47c494bb602d886ce11e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } }
I20260927 14:57:17.368263 7769 raft_consensus.cc:348] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "280b4c9e2726475989c4e06525b89aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 } } peers { permanent_uuid: "5e76f2b076e14fcca79b8699d6be4704" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35207 } } peers { permanent_uuid: "1544dca4984d47c494bb602d886ce11e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } }
I20260927 14:57:17.368530 7768 raft_consensus.cc:374] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 5e76f2b076e14fcca79b8699d6be4704 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:17.368541 7769 raft_consensus.cc:374] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:17.368592 7769 raft_consensus.cc:729] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 280b4c9e2726475989c4e06525b89aec, State: Initialized, Role: FOLLOWER
I20260927 14:57:17.368592 7768 raft_consensus.cc:729] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 5e76f2b076e14fcca79b8699d6be4704 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e76f2b076e14fcca79b8699d6be4704, State: Initialized, Role: FOLLOWER
I20260927 14:57:17.369060 7768 consensus_queue.cc:260] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 5e76f2b076e14fcca79b8699d6be4704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "280b4c9e2726475989c4e06525b89aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 } } peers { permanent_uuid: "5e76f2b076e14fcca79b8699d6be4704" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35207 } } peers { permanent_uuid: "1544dca4984d47c494bb602d886ce11e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } }
I20260927 14:57:17.369071 7769 consensus_queue.cc:260] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "280b4c9e2726475989c4e06525b89aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 } } peers { permanent_uuid: "5e76f2b076e14fcca79b8699d6be4704" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35207 } } peers { permanent_uuid: "1544dca4984d47c494bb602d886ce11e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } }
I20260927 14:57:17.370199 7769 ts_tablet_manager.cc:1428] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 14:57:17.370414 7734 heartbeater.cc:503] Master 127.0.98.62:44545 was elected leader, sending a full tablet report...
I20260927 14:57:17.371740 7768 ts_tablet_manager.cc:1428] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 5e76f2b076e14fcca79b8699d6be4704: Time spent starting tablet: real 0.008s user 0.001s sys 0.006s
I20260927 14:57:17.372028 7597 heartbeater.cc:503] Master 127.0.98.62:44545 was elected leader, sending a full tablet report...
I20260927 14:57:17.374845 7774 tablet_bootstrap.cc:492] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 1544dca4984d47c494bb602d886ce11e: Bootstrap starting.
I20260927 14:57:17.377691 7774 tablet_bootstrap.cc:654] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 1544dca4984d47c494bb602d886ce11e: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:17.378517 7774 log.cc:824] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 1544dca4984d47c494bb602d886ce11e: Log is configured to *not* fsync() on all Append() calls
I20260927 14:57:17.380532 7774 tablet_bootstrap.cc:492] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 1544dca4984d47c494bb602d886ce11e: No bootstrap required, opened a new log
I20260927 14:57:17.380739 7774 ts_tablet_manager.cc:1397] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 1544dca4984d47c494bb602d886ce11e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 14:57:17.383735 7774 raft_consensus.cc:348] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 1544dca4984d47c494bb602d886ce11e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "280b4c9e2726475989c4e06525b89aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 } } peers { permanent_uuid: "5e76f2b076e14fcca79b8699d6be4704" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35207 } } peers { permanent_uuid: "1544dca4984d47c494bb602d886ce11e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } }
I20260927 14:57:17.384011 7774 raft_consensus.cc:374] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 1544dca4984d47c494bb602d886ce11e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:17.384070 7774 raft_consensus.cc:729] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 1544dca4984d47c494bb602d886ce11e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1544dca4984d47c494bb602d886ce11e, State: Initialized, Role: FOLLOWER
I20260927 14:57:17.384507 7774 consensus_queue.cc:260] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 1544dca4984d47c494bb602d886ce11e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "280b4c9e2726475989c4e06525b89aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 } } peers { permanent_uuid: "5e76f2b076e14fcca79b8699d6be4704" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35207 } } peers { permanent_uuid: "1544dca4984d47c494bb602d886ce11e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } }
I20260927 14:57:17.385164 7460 heartbeater.cc:503] Master 127.0.98.62:44545 was elected leader, sending a full tablet report...
I20260927 14:57:17.385582 7774 ts_tablet_manager.cc:1428] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 1544dca4984d47c494bb602d886ce11e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 14:57:17.390434 7772 raft_consensus.cc:482] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:57:17.390647 7772 raft_consensus.cc:504] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "280b4c9e2726475989c4e06525b89aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 } } peers { permanent_uuid: "5e76f2b076e14fcca79b8699d6be4704" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35207 } } peers { permanent_uuid: "1544dca4984d47c494bb602d886ce11e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } }
I20260927 14:57:17.391839 7772 leader_election.cc:288] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e76f2b076e14fcca79b8699d6be4704 (127.0.98.2:35207), 1544dca4984d47c494bb602d886ce11e (127.0.98.1:34875)
I20260927 14:57:17.396714 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.392307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36037 (local address 127.0.98.2:35207)
0927 14:57:17.392472 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:17.392476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:17.392490 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:57:17.392543 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:17.392547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:17.392597 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:17.392679 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:17.393021 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.393826 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.395453 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.396076 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.396177 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.396244 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:17.396501 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 14:57:17.396577 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 14:57:17.396714 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.392224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41403 (local address 127.0.98.1:34875)
0927 14:57:17.392349 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:17.392353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:17.392493 (+ 140us) server_negotiation.cc:407] Connection header received
0927 14:57:17.392552 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:17.392554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:17.392597 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:17.392679 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:17.393021 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.393826 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.395453 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.396079 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.396177 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.396245 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:17.396519 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 14:57:17.396586 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 14:57:17.396733 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.392109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34875 (local address 127.0.98.3:41403)
0927 14:57:17.392418 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:17.392429 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:17.392490 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:17.392694 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:17.392697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:17.392723 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:17.392887 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:17.392894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.393929 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.393933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:17.395342 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:17.395353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.396204 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.396208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:17.396366 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.396403 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 14:57:17.396478 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 14:57:17.396577 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":339,"spinlock_wait_cycles":8832}
I20260927 14:57:17.396733 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.392105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35207 (local address 127.0.98.3:36037)
0927 14:57:17.392215 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:17.392229 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:17.392322 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:17.392694 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:17.392697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:17.392716 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:17.392887 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:17.392894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.393929 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.393933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:17.395342 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:17.395353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.396204 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.396208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:17.396361 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.396403 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 14:57:17.396478 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 14:57:17.396577 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":129,"spinlock_wait_cycles":768}
I20260927 14:57:17.397521 7413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb37150bd2ec4141b6da2e66c2a5a6d1" candidate_uuid: "280b4c9e2726475989c4e06525b89aec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1544dca4984d47c494bb602d886ce11e" is_pre_election: true
I20260927 14:57:17.397718 7550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb37150bd2ec4141b6da2e66c2a5a6d1" candidate_uuid: "280b4c9e2726475989c4e06525b89aec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e76f2b076e14fcca79b8699d6be4704" is_pre_election: true
I20260927 14:57:17.397862 7413 raft_consensus.cc:2454] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 1544dca4984d47c494bb602d886ce11e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 280b4c9e2726475989c4e06525b89aec in term 0.
I20260927 14:57:17.398001 7550 raft_consensus.cc:2454] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 5e76f2b076e14fcca79b8699d6be4704 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 280b4c9e2726475989c4e06525b89aec in term 0.
I20260927 14:57:17.398368 7620 leader_election.cc:302] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1544dca4984d47c494bb602d886ce11e, 280b4c9e2726475989c4e06525b89aec, 5e76f2b076e14fcca79b8699d6be4704; no voters:
I20260927 14:57:17.398705 7772 raft_consensus.cc:2781] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:57:17.398828 7772 raft_consensus.cc:482] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:57:17.398878 7772 raft_consensus.cc:3037] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:17.399906 7772 raft_consensus.cc:504] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "280b4c9e2726475989c4e06525b89aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 } } peers { permanent_uuid: "5e76f2b076e14fcca79b8699d6be4704" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35207 } } peers { permanent_uuid: "1544dca4984d47c494bb602d886ce11e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } }
I20260927 14:57:17.400272 7772 leader_election.cc:288] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec [CANDIDATE]: Term 1 election: Requested vote from peers 5e76f2b076e14fcca79b8699d6be4704 (127.0.98.2:35207), 1544dca4984d47c494bb602d886ce11e (127.0.98.1:34875)
I20260927 14:57:17.400614 7550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb37150bd2ec4141b6da2e66c2a5a6d1" candidate_uuid: "280b4c9e2726475989c4e06525b89aec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e76f2b076e14fcca79b8699d6be4704"
I20260927 14:57:17.400678 7413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb37150bd2ec4141b6da2e66c2a5a6d1" candidate_uuid: "280b4c9e2726475989c4e06525b89aec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1544dca4984d47c494bb602d886ce11e"
I20260927 14:57:17.400732 7550 raft_consensus.cc:3037] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 5e76f2b076e14fcca79b8699d6be4704 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:17.400795 7413 raft_consensus.cc:3037] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 1544dca4984d47c494bb602d886ce11e [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:17.401749 7413 raft_consensus.cc:2454] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 1544dca4984d47c494bb602d886ce11e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 280b4c9e2726475989c4e06525b89aec in term 1.
I20260927 14:57:17.401762 7550 raft_consensus.cc:2454] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 5e76f2b076e14fcca79b8699d6be4704 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 280b4c9e2726475989c4e06525b89aec in term 1.
I20260927 14:57:17.402034 7623 leader_election.cc:302] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 280b4c9e2726475989c4e06525b89aec, 5e76f2b076e14fcca79b8699d6be4704; no voters:
I20260927 14:57:17.402189 7772 raft_consensus.cc:2781] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:57:17.402415 7772 raft_consensus.cc:686] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec [term 1 LEADER]: Becoming Leader. State: Replica: 280b4c9e2726475989c4e06525b89aec, State: Running, Role: LEADER
I20260927 14:57:17.402782 7772 consensus_queue.cc:237] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "280b4c9e2726475989c4e06525b89aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 } } peers { permanent_uuid: "5e76f2b076e14fcca79b8699d6be4704" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35207 } } peers { permanent_uuid: "1544dca4984d47c494bb602d886ce11e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } }
I20260927 14:57:17.406571 6481 catalog_manager.cc:5495] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec reported cstate change: term changed from 0 to 1, leader changed from <none> to 280b4c9e2726475989c4e06525b89aec (127.0.98.3). New cstate: current_term: 1 leader_uuid: "280b4c9e2726475989c4e06525b89aec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "280b4c9e2726475989c4e06525b89aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e76f2b076e14fcca79b8699d6be4704" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1544dca4984d47c494bb602d886ce11e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } health_report { overall_health: UNKNOWN } } }
I20260927 14:57:17.422614 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.419184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38164 (local address 127.0.98.3:42813)
0927 14:57:17.419286 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:17.419289 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:17.419302 (+ 13us) server_negotiation.cc:407] Connection header received
0927 14:57:17.419399 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:17.419401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:17.419455 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:17.419552 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:57:17.419799 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.420332 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.421229 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.421485 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.421567 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.421816 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:57:17.422279 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:57:17.422367 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:17.422422 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 14:57:17.422483 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 14:57:17.428572 7413 raft_consensus.cc:1261] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 1544dca4984d47c494bb602d886ce11e [term 1 FOLLOWER]: Refusing update from remote peer 280b4c9e2726475989c4e06525b89aec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:57:17.428572 7550 raft_consensus.cc:1261] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 5e76f2b076e14fcca79b8699d6be4704 [term 1 FOLLOWER]: Refusing update from remote peer 280b4c9e2726475989c4e06525b89aec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:57:17.429437 7772 consensus_queue.cc:1035] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "1544dca4984d47c494bb602d886ce11e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:57:17.429651 7777 consensus_queue.cc:1035] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e76f2b076e14fcca79b8699d6be4704" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:57:17.439559 7781 mvcc.cc:205] Tried to move back new op lower bound from 7333974169299546112 to 7333974169204727808. Current Snapshot: MvccSnapshot[applied={T|T < 7333974169299546112}]
I20260927 14:57:17.443499 7782 mvcc.cc:205] Tried to move back new op lower bound from 7333974169299546112 to 7333974169204727808. Current Snapshot: MvccSnapshot[applied={T|T < 7333974169299546112}]
I20260927 14:57:17.449451 6481 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:57:17.449671 6481 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:57:17.454015 6481 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e1e4b37da644604875d93c9bf1060ba: Sending DeleteTablet for 3 replicas of tablet eb37150bd2ec4141b6da2e66c2a5a6d1
I20260927 14:57:17.454985 7667 tablet_service.cc:1514] Processing DeleteTablet for tablet eb37150bd2ec4141b6da2e66c2a5a6d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:57:17 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38160
I20260927 14:57:17.454993 7530 tablet_service.cc:1514] Processing DeleteTablet for tablet eb37150bd2ec4141b6da2e66c2a5a6d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:57:17 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34210
I20260927 14:57:17.455024 7393 tablet_service.cc:1514] Processing DeleteTablet for tablet eb37150bd2ec4141b6da2e66c2a5a6d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:57:17 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56296
I20260927 14:57:17.455542 7792 tablet_replica.cc:330] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 1544dca4984d47c494bb602d886ce11e: stopping tablet replica
I20260927 14:57:17.455737 7791 tablet_replica.cc:330] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec: stopping tablet replica
I20260927 14:57:17.455900 7792 raft_consensus.cc:2229] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 1544dca4984d47c494bb602d886ce11e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:57:17.456138 7791 raft_consensus.cc:2229] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:57:17.456247 7792 raft_consensus.cc:2258] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 1544dca4984d47c494bb602d886ce11e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:57:17.456611 7791 raft_consensus.cc:2258] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:57:17.457373 7790 tablet_replica.cc:330] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 5e76f2b076e14fcca79b8699d6be4704: stopping tablet replica
I20260927 14:57:17.457748 7790 raft_consensus.cc:2229] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 5e76f2b076e14fcca79b8699d6be4704 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:57:17.458087 7790 raft_consensus.cc:2258] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 5e76f2b076e14fcca79b8699d6be4704 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:57:17.458524 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 6456
I20260927 14:57:17.459558 7792 ts_tablet_manager.cc:1905] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 1544dca4984d47c494bb602d886ce11e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:57:17.460855 7791 ts_tablet_manager.cc:1905] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:57:17.462301 7792 ts_tablet_manager.cc:1918] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 1544dca4984d47c494bb602d886ce11e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:57:17.462420 7792 log.cc:1196] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 1544dca4984d47c494bb602d886ce11e: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-0/wal/wals/eb37150bd2ec4141b6da2e66c2a5a6d1
I20260927 14:57:17.462736 7792 ts_tablet_manager.cc:1939] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 1544dca4984d47c494bb602d886ce11e: Deleting consensus metadata
I20260927 14:57:17.463685 7791 ts_tablet_manager.cc:1918] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:57:17.463778 7791 log.cc:1196] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/wal/wals/eb37150bd2ec4141b6da2e66c2a5a6d1
I20260927 14:57:17.464124 7791 ts_tablet_manager.cc:1939] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 280b4c9e2726475989c4e06525b89aec: Deleting consensus metadata
I20260927 14:57:17.466478 7790 ts_tablet_manager.cc:1905] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 5e76f2b076e14fcca79b8699d6be4704: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 14:57:17.466810 7349 connection.cc:535] server connection from 127.0.0.1:56296 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 14:57:17.466970 7620 connection.cc:535] server connection from 127.0.0.1:38160 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 14:57:17.469190 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.457157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42770 (local address 127.0.98.61:36483)
0927 14:57:17.457271 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:17.457275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:17.457290 (+ 15us) server_negotiation.cc:407] Connection header received
0927 14:57:17.457330 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:17.457333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:17.457381 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:17.457461 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:17.458775 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.459564 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.460924 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.461248 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.461329 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.463586 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:17.463590 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:17.463593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:17.463624 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:17.465568 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:17.466504 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:17.466521 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:17.466522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:17.466617 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:17.467462 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:17.467465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:17.467467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:17.468125 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:17.468399 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:17.468869 (+ 470us) server_negotiation.cc:299] Negotiation successful
0927 14:57:17.469030 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 14:57:17.470371 7790 ts_tablet_manager.cc:1918] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 5e76f2b076e14fcca79b8699d6be4704: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:57:17.470472 7790 log.cc:1196] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 5e76f2b076e14fcca79b8699d6be4704: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/wal/wals/eb37150bd2ec4141b6da2e66c2a5a6d1
I20260927 14:57:17.470844 7790 ts_tablet_manager.cc:1939] T eb37150bd2ec4141b6da2e66c2a5a6d1 P 5e76f2b076e14fcca79b8699d6be4704: Deleting consensus metadata
W20260927 14:57:17.472105 7486 connection.cc:441] server connection from 127.0.0.1:34210 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34210 (request call id 1) could send its response
I20260927 14:57:17.481366 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.456775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35646 (local address 127.0.98.60:41351)
0927 14:57:17.456884 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:17.456888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:17.471119 (+ 14231us) server_negotiation.cc:407] Connection header received
0927 14:57:17.471293 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:17.471297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:17.471348 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:17.471412 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:17.472415 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.473164 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.474343 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.475177 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.475256 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.477392 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:17.477395 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:17.477398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:17.477421 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:17.479044 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:17.479655 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:17.479658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:17.479661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:17.479850 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:17.480146 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:17.480149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:17.480151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:17.480356 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:17.480438 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:17.481063 (+ 625us) server_negotiation.cc:299] Negotiation successful
0927 14:57:17.481200 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":495,"server-negotiator.queue_time_us":42}
I20260927 14:57:17.481366 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.471181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42776 (local address 127.0.98.61:36483)
0927 14:57:17.471286 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:17.471290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:17.471966 (+ 676us) server_negotiation.cc:407] Connection header received
0927 14:57:17.472128 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:17.472132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:17.472173 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:17.472221 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:17.473079 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.473856 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.475131 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.475434 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.475524 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.477210 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:17.477213 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:17.477217 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:17.477244 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:17.479123 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:17.479655 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:17.479658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:17.479661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:17.479755 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:17.480146 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:17.480149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:17.480151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:17.480597 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:17.480704 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:17.481087 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 14:57:17.481199 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 14:57:17.482053 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:17.472788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35648 (local address 127.0.98.60:41351)
0927 14:57:17.472891 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:17.472895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:17.473181 (+ 286us) server_negotiation.cc:407] Connection header received
0927 14:57:17.473329 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:17.473332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:17.473378 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:17.473424 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:17.474368 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.475000 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.476203 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:17.476565 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:17.476632 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:17.478468 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:17.478472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:17.478475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:17.478500 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:17.480221 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:17.480680 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:17.480683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:17.480685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:17.480763 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:17.481041 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:17.481044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:17.481046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:17.481249 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:17.481344 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:17.481785 (+ 441us) server_negotiation.cc:299] Negotiation successful
0927 14:57:17.481897 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":41}
I20260927 14:57:18.443290 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:18.442722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.98.2:54877)
0927 14:57:18.443068 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:18.443171 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":125,"threads_started":1}
W20260927 14:57:18.444039 7597 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44545 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
I20260927 14:57:18.444541 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:18.443876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.98.3:54147)
0927 14:57:18.444347 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:18.444437 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":89,"threads_started":1}
W20260927 14:57:18.444803 7734 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44545 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
I20260927 14:57:18.445075 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:18.444677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.98.1:32891)
0927 14:57:18.444921 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:18.445000 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":63,"threads_started":1}
W20260927 14:57:18.445339 7460 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44545 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
I20260927 14:57:18.954244 7803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3e1e4b37da644604875d93c9bf1060ba)
I20260927 14:57:18.954469 7803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: false } }
I20260927 14:57:18.955202 7804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3e1e4b37da644604875d93c9bf1060ba)
I20260927 14:57:18.955394 7804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: false } }
I20260927 14:57:18.955592 7803 leader_election.cc:288] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e1e4b37da644604875d93c9bf1060ba (127.0.98.62:44545), 86c641e5060c4fe88d0b39cf0550cd6f (127.0.98.61:36483)
I20260927 14:57:18.956252 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:18.955758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.0.1:37034)
0927 14:57:18.956052 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:18.956146 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":49,"threads_started":1}
I20260927 14:57:18.956386 7804 leader_election.cc:288] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e1e4b37da644604875d93c9bf1060ba (127.0.98.62:44545), 56ffb5d4c52e4a5fa29276ae2214dc99 (127.0.98.60:41351)
W20260927 14:57:18.956781 7267 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
I20260927 14:57:18.957109 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:18.956465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.0.1:37038)
0927 14:57:18.956926 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:18.957001 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":82,"threads_started":1}
W20260927 14:57:18.957269 7077 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
I20260927 14:57:18.958168 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:18.957900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.0.1:37050)
0927 14:57:18.958046 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:18.958103 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 14:57:18.958175 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:18.957906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.0.1:37060)
0927 14:57:18.958043 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:18.958091 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260927 14:57:18.958293 7267 leader_election.cc:334] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3e1e4b37da644604875d93c9bf1060ba (127.0.98.62:44545): Network error: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
W20260927 14:57:18.958312 7077 leader_election.cc:334] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3e1e4b37da644604875d93c9bf1060ba (127.0.98.62:44545): Network error: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
I20260927 14:57:18.960191 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:18.956086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36483 (local address 127.0.0.1:55192)
0927 14:57:18.956262 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:18.956274 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:18.956441 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:18.956868 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:18.956871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:18.956890 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:18.956989 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:18.956996 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:18.957988 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:18.957991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:18.959084 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:18.959094 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:18.959779 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:18.959782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:18.959872 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:18.959898 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 14:57:18.959966 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 14:57:18.960058 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":2,"thread_start_us":53,"threads_started":1}
I20260927 14:57:18.960191 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:18.955887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55192 (local address 127.0.98.61:36483)
0927 14:57:18.956144 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:18.956149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:18.956357 (+ 208us) server_negotiation.cc:407] Connection header received
0927 14:57:18.956610 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:18.956613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:18.956676 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:18.956750 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:18.957115 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:18.957860 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:18.959197 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:18.959668 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:18.959768 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:18.959830 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:18.959987 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 14:57:18.960061 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":55,"threads_started":1}
I20260927 14:57:18.960835 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:18.956727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33386 (local address 127.0.98.60:41351)
0927 14:57:18.957320 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:18.957324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:18.957337 (+ 13us) server_negotiation.cc:407] Connection header received
0927 14:57:18.957375 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:18.957378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:18.957421 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:18.957502 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:18.957756 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:18.958691 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:18.959873 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:18.960337 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:18.960434 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:18.960494 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:18.960643 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 14:57:18.960717 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":530,"thread_start_us":55,"threads_started":1}
I20260927 14:57:18.960893 7100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" is_pre_election: true
I20260927 14:57:18.961113 7100 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56ffb5d4c52e4a5fa29276ae2214dc99 in term 1.
I20260927 14:57:18.961359 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:18.956618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41351 (local address 127.0.0.1:33386)
0927 14:57:18.956973 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:18.956985 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:18.957066 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:18.957521 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:18.957524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:18.957540 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:18.957632 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:18.957638 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:18.958874 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:18.958877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:18.959776 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:18.959785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:18.960435 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:18.960444 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:18.960528 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:18.960549 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 14:57:18.960622 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 14:57:18.960691 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":61,"threads_started":1}
I20260927 14:57:18.961440 7266 leader_election.cc:302] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 56ffb5d4c52e4a5fa29276ae2214dc99, 86c641e5060c4fe88d0b39cf0550cd6f; no voters: 3e1e4b37da644604875d93c9bf1060ba
I20260927 14:57:18.961678 7803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 14:57:18.961741 7803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3e1e4b37da644604875d93c9bf1060ba)
I20260927 14:57:18.961727 7289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" is_pre_election: true
I20260927 14:57:18.961789 7803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [term 1 FOLLOWER]: Advancing to term 2
I20260927 14:57:18.963019 7803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: false } }
I20260927 14:57:18.963279 7289 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 86c641e5060c4fe88d0b39cf0550cd6f in current term 2: Already voted for candidate 56ffb5d4c52e4a5fa29276ae2214dc99 in this term.
I20260927 14:57:18.963439 7803 leader_election.cc:288] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [CANDIDATE]: Term 2 election: Requested vote from peers 3e1e4b37da644604875d93c9bf1060ba (127.0.98.62:44545), 86c641e5060c4fe88d0b39cf0550cd6f (127.0.98.61:36483)
I20260927 14:57:18.963825 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:18.963576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.0.1:37072)
0927 14:57:18.963698 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:18.963753 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 14:57:18.963838 7100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "86c641e5060c4fe88d0b39cf0550cd6f"
I20260927 14:57:18.963963 7074 leader_election.cc:302] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86c641e5060c4fe88d0b39cf0550cd6f; no voters: 3e1e4b37da644604875d93c9bf1060ba, 56ffb5d4c52e4a5fa29276ae2214dc99
I20260927 14:57:18.963968 7100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [term 1 FOLLOWER]: Advancing to term 2
I20260927 14:57:18.964511 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:18.964302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.0.1:37082)
0927 14:57:18.964407 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:18.964459 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 14:57:18.964609 7267 leader_election.cc:334] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3e1e4b37da644604875d93c9bf1060ba (127.0.98.62:44545): Network error: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
I20260927 14:57:18.965191 7100 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56ffb5d4c52e4a5fa29276ae2214dc99 in term 2.
I20260927 14:57:18.965297 7804 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 14:57:18.965441 7266 leader_election.cc:302] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 56ffb5d4c52e4a5fa29276ae2214dc99, 86c641e5060c4fe88d0b39cf0550cd6f; no voters: 3e1e4b37da644604875d93c9bf1060ba
I20260927 14:57:18.965606 7803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 14:57:18.965822 7803 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [term 2 LEADER]: Becoming Leader. State: Replica: 56ffb5d4c52e4a5fa29276ae2214dc99, State: Running, Role: LEADER
I20260927 14:57:18.966063 7803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: false } }
I20260927 14:57:18.967104 7813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56ffb5d4c52e4a5fa29276ae2214dc99. Latest consensus state: current_term: 2 leader_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: false } } }
I20260927 14:57:18.967224 7813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:18.967501 7815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:57:18.969898 7815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:57:18.970361 7815 catalog_manager.cc:1259] Loaded cluster ID: 2dae8741e60142f9ba09d6c7b3a0d359
I20260927 14:57:18.970510 7815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:57:18.971356 7815 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:57:18.971772 7815 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99: Loaded TSK: 0
I20260927 14:57:18.972470 7815 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:57:18.991602 7276 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:57:18.994774 7276 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:57:18.997851 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:18.997602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.0.1:37094)
0927 14:57:18.997715 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:18.997773 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 14:57:18.997946 7100 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [term 2 FOLLOWER]: Refusing update from remote peer 56ffb5d4c52e4a5fa29276ae2214dc99: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 14:57:18.998530 7813 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 14:57:18.998617 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:18.998368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.0.1:37098)
0927 14:57:18.998476 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:18.998535 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 14:57:18.998786 7267 consensus_peers.cc:487] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 -> Peer 3e1e4b37da644604875d93c9bf1060ba (127.0.98.62:44545): Couldn't send request to peer 3e1e4b37da644604875d93c9bf1060ba. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 14:57:19.000355 7804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56ffb5d4c52e4a5fa29276ae2214dc99. Latest consensus state: current_term: 2 leader_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: false } } }
I20260927 14:57:19.000501 7804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:57:19.001621 7803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: false } } }
I20260927 14:57:19.001736 7803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99 [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:19.001893 7817 mvcc.cc:205] Tried to move back new op lower bound from 7333974175730724864 to 7333974175607316480. Current Snapshot: MvccSnapshot[applied={T|T < 7333974175730724864}]
I20260927 14:57:19.002061 7804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3e1e4b37da644604875d93c9bf1060ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } peers { permanent_uuid: "86c641e5060c4fe88d0b39cf0550cd6f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36483 } attrs { promote: false } } peers { permanent_uuid: "56ffb5d4c52e4a5fa29276ae2214dc99" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41351 } attrs { promote: false } } }
I20260927 14:57:19.002172 7804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86c641e5060c4fe88d0b39cf0550cd6f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:57:19.014062 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:19.009474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35207 (local address 127.0.0.1:42472)
0927 14:57:19.009562 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:19.009576 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:19.009650 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:19.010107 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:19.010110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:19.010126 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:19.010222 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:19.010229 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:19.011383 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:19.011387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:19.012690 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:19.012703 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:19.013589 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:19.013592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:19.013728 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:19.013756 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 14:57:19.013820 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 14:57:19.013906 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":129}
I20260927 14:57:19.014204 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:19.009015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42813 (local address 127.0.0.1:60782)
0927 14:57:19.009244 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:19.009265 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:19.009361 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:19.010177 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:19.010180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:19.010198 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:19.010273 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:19.010277 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:19.011383 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:19.011387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:19.012996 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:19.013007 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:19.013861 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:19.013864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:19.013939 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:19.013958 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 14:57:19.014022 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 14:57:19.014111 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":163}
I20260927 14:57:19.014539 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:19.009211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60782 (local address 127.0.98.3:42813)
0927 14:57:19.009915 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:19.009919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:19.009934 (+ 15us) server_negotiation.cc:407] Connection header received
0927 14:57:19.009984 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:19.009987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:19.010036 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:19.010367 (+ 331us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:19.010475 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:19.011252 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:19.013119 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:19.013731 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:19.013866 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:19.013931 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:19.014038 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 14:57:19.014122 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":82,"threads_started":1}
I20260927 14:57:19.014639 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:19.009608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42472 (local address 127.0.98.2:35207)
0927 14:57:19.009866 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:19.009870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:19.009883 (+ 13us) server_negotiation.cc:407] Connection header received
0927 14:57:19.009928 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:19.009931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:19.009980 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:19.010061 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:19.010535 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:19.011271 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:19.012842 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:19.013462 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:19.013564 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:19.013636 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:19.014401 (+ 765us) server_negotiation.cc:299] Negotiation successful
0927 14:57:19.014496 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":85,"threads_started":1}
I20260927 14:57:19.014962 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:19.009586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34875 (local address 127.0.0.1:40426)
0927 14:57:19.010261 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:19.010274 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:19.010358 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:19.010569 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:19.010572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:19.010588 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:19.010660 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:19.010666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:19.012090 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:19.012095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:19.013709 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:19.013717 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:19.014565 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:19.014568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:19.014653 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:19.014678 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 14:57:19.014745 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 14:57:19.014857 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":18,"thread_start_us":108,"threads_started":1}
I20260927 14:57:19.015319 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:19.009902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40426 (local address 127.0.98.1:34875)
0927 14:57:19.010152 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:19.010157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:19.010350 (+ 193us) server_negotiation.cc:407] Connection header received
0927 14:57:19.010433 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:19.010436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:19.010484 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:19.010572 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:19.010761 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:19.011906 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:19.013843 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:19.014448 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:19.014552 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:19.014620 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:19.015120 (+ 500us) server_negotiation.cc:299] Negotiation successful
0927 14:57:19.015216 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":91,"threads_started":1}
I20260927 14:57:19.015264 7530 tablet_service.cc:1467] Processing CreateTablet for tablet 300a4a7704114716b98a41830284d499 (DEFAULT_TABLE table=test-table [id=2f401691a8354232818a4a5087cb70fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:57:19.015367 7667 tablet_service.cc:1467] Processing CreateTablet for tablet 300a4a7704114716b98a41830284d499 (DEFAULT_TABLE table=test-table [id=2f401691a8354232818a4a5087cb70fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:57:19.015573 7530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 300a4a7704114716b98a41830284d499. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:19.015602 7667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 300a4a7704114716b98a41830284d499. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:19.018260 7826 tablet_bootstrap.cc:492] T 300a4a7704114716b98a41830284d499 P 280b4c9e2726475989c4e06525b89aec: Bootstrap starting.
I20260927 14:57:19.018692 7393 tablet_service.cc:1467] Processing CreateTablet for tablet 300a4a7704114716b98a41830284d499 (DEFAULT_TABLE table=test-table [id=2f401691a8354232818a4a5087cb70fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:57:19.018987 7393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 300a4a7704114716b98a41830284d499. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:19.019001 7827 tablet_bootstrap.cc:492] T 300a4a7704114716b98a41830284d499 P 5e76f2b076e14fcca79b8699d6be4704: Bootstrap starting.
I20260927 14:57:19.019378 7826 tablet_bootstrap.cc:654] T 300a4a7704114716b98a41830284d499 P 280b4c9e2726475989c4e06525b89aec: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:19.019845 7827 tablet_bootstrap.cc:654] T 300a4a7704114716b98a41830284d499 P 5e76f2b076e14fcca79b8699d6be4704: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:19.020798 7826 tablet_bootstrap.cc:492] T 300a4a7704114716b98a41830284d499 P 280b4c9e2726475989c4e06525b89aec: No bootstrap required, opened a new log
I20260927 14:57:19.020892 7826 ts_tablet_manager.cc:1397] T 300a4a7704114716b98a41830284d499 P 280b4c9e2726475989c4e06525b89aec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 14:57:19.021279 7830 tablet_bootstrap.cc:492] T 300a4a7704114716b98a41830284d499 P 1544dca4984d47c494bb602d886ce11e: Bootstrap starting.
I20260927 14:57:19.021472 7826 raft_consensus.cc:348] T 300a4a7704114716b98a41830284d499 P 280b4c9e2726475989c4e06525b89aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "280b4c9e2726475989c4e06525b89aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 } } peers { permanent_uuid: "1544dca4984d47c494bb602d886ce11e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "5e76f2b076e14fcca79b8699d6be4704" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35207 } }
I20260927 14:57:19.021576 7826 raft_consensus.cc:374] T 300a4a7704114716b98a41830284d499 P 280b4c9e2726475989c4e06525b89aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:19.021607 7826 raft_consensus.cc:729] T 300a4a7704114716b98a41830284d499 P 280b4c9e2726475989c4e06525b89aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 280b4c9e2726475989c4e06525b89aec, State: Initialized, Role: FOLLOWER
I20260927 14:57:19.021759 7826 consensus_queue.cc:260] T 300a4a7704114716b98a41830284d499 P 280b4c9e2726475989c4e06525b89aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "280b4c9e2726475989c4e06525b89aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 } } peers { permanent_uuid: "1544dca4984d47c494bb602d886ce11e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "5e76f2b076e14fcca79b8699d6be4704" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35207 } }
I20260927 14:57:19.022117 7826 ts_tablet_manager.cc:1428] T 300a4a7704114716b98a41830284d499 P 280b4c9e2726475989c4e06525b89aec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:57:19.022282 7830 tablet_bootstrap.cc:654] T 300a4a7704114716b98a41830284d499 P 1544dca4984d47c494bb602d886ce11e: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:19.022292 7827 tablet_bootstrap.cc:492] T 300a4a7704114716b98a41830284d499 P 5e76f2b076e14fcca79b8699d6be4704: No bootstrap required, opened a new log
I20260927 14:57:19.022377 7827 ts_tablet_manager.cc:1397] T 300a4a7704114716b98a41830284d499 P 5e76f2b076e14fcca79b8699d6be4704: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 14:57:19.022861 7827 raft_consensus.cc:348] T 300a4a7704114716b98a41830284d499 P 5e76f2b076e14fcca79b8699d6be4704 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "280b4c9e2726475989c4e06525b89aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 } } peers { permanent_uuid: "1544dca4984d47c494bb602d886ce11e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "5e76f2b076e14fcca79b8699d6be4704" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35207 } }
I20260927 14:57:19.022982 7827 raft_consensus.cc:374] T 300a4a7704114716b98a41830284d499 P 5e76f2b076e14fcca79b8699d6be4704 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:19.023013 7827 raft_consensus.cc:729] T 300a4a7704114716b98a41830284d499 P 5e76f2b076e14fcca79b8699d6be4704 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e76f2b076e14fcca79b8699d6be4704, State: Initialized, Role: FOLLOWER
I20260927 14:57:19.023088 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:19.022589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.98.3:49059)
0927 14:57:19.022913 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:19.022988 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":93,"threads_started":1}
I20260927 14:57:19.023165 7827 consensus_queue.cc:260] T 300a4a7704114716b98a41830284d499 P 5e76f2b076e14fcca79b8699d6be4704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "280b4c9e2726475989c4e06525b89aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 } } peers { permanent_uuid: "1544dca4984d47c494bb602d886ce11e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "5e76f2b076e14fcca79b8699d6be4704" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35207 } }
I20260927 14:57:19.023492 7827 ts_tablet_manager.cc:1428] T 300a4a7704114716b98a41830284d499 P 5e76f2b076e14fcca79b8699d6be4704: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:57:19.023897 7830 tablet_bootstrap.cc:492] T 300a4a7704114716b98a41830284d499 P 1544dca4984d47c494bb602d886ce11e: No bootstrap required, opened a new log
I20260927 14:57:19.024006 7830 ts_tablet_manager.cc:1397] T 300a4a7704114716b98a41830284d499 P 1544dca4984d47c494bb602d886ce11e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 14:57:19.024431 7830 raft_consensus.cc:348] T 300a4a7704114716b98a41830284d499 P 1544dca4984d47c494bb602d886ce11e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "280b4c9e2726475989c4e06525b89aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 } } peers { permanent_uuid: "1544dca4984d47c494bb602d886ce11e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "5e76f2b076e14fcca79b8699d6be4704" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35207 } }
I20260927 14:57:19.024540 7830 raft_consensus.cc:374] T 300a4a7704114716b98a41830284d499 P 1544dca4984d47c494bb602d886ce11e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:19.024575 7830 raft_consensus.cc:729] T 300a4a7704114716b98a41830284d499 P 1544dca4984d47c494bb602d886ce11e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1544dca4984d47c494bb602d886ce11e, State: Initialized, Role: FOLLOWER
I20260927 14:57:19.024947 7733 heartbeater.cc:503] Master 127.0.98.60:41351 was elected leader, sending a full tablet report...
W20260927 14:57:19.025339 7736 tablet.cc:2369] T 300a4a7704114716b98a41830284d499 P 280b4c9e2726475989c4e06525b89aec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:57:19.026144 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:19.025861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.98.3:54517)
0927 14:57:19.026008 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:19.026064 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 14:57:19.026907 7830 consensus_queue.cc:260] T 300a4a7704114716b98a41830284d499 P 1544dca4984d47c494bb602d886ce11e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "280b4c9e2726475989c4e06525b89aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 } } peers { permanent_uuid: "1544dca4984d47c494bb602d886ce11e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "5e76f2b076e14fcca79b8699d6be4704" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35207 } }
I20260927 14:57:19.027194 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:19.023927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.98.2:60075)
0927 14:57:19.027024 (+ 3097us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:19.027087 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3026,"thread_start_us":89,"threads_started":1}
I20260927 14:57:19.027288 7830 ts_tablet_manager.cc:1428] T 300a4a7704114716b98a41830284d499 P 1544dca4984d47c494bb602d886ce11e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260927 14:57:19.027879 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:19.027622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.98.2:33153)
0927 14:57:19.027748 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:19.027807 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 14:57:19.028105 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:19.027706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.98.1:48563)
0927 14:57:19.027940 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:19.027997 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":104,"threads_started":1}
I20260927 14:57:19.028797 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:19.028567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.98.1:34393)
0927 14:57:19.028669 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:19.028720 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 14:57:19.029043 7459 heartbeater.cc:503] Master 127.0.98.60:41351 was elected leader, sending a full tablet report...
W20260927 14:57:19.056150 7599 tablet.cc:2369] T 300a4a7704114716b98a41830284d499 P 5e76f2b076e14fcca79b8699d6be4704: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 14:57:19.097294 7462 tablet.cc:2369] T 300a4a7704114716b98a41830284d499 P 1544dca4984d47c494bb602d886ce11e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:57:19.176522 7836 raft_consensus.cc:482] T 300a4a7704114716b98a41830284d499 P 1544dca4984d47c494bb602d886ce11e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:57:19.176694 7836 raft_consensus.cc:504] T 300a4a7704114716b98a41830284d499 P 1544dca4984d47c494bb602d886ce11e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "280b4c9e2726475989c4e06525b89aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 } } peers { permanent_uuid: "1544dca4984d47c494bb602d886ce11e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "5e76f2b076e14fcca79b8699d6be4704" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35207 } }
I20260927 14:57:19.177754 7836 leader_election.cc:288] T 300a4a7704114716b98a41830284d499 P 1544dca4984d47c494bb602d886ce11e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 280b4c9e2726475989c4e06525b89aec (127.0.98.3:42813), 5e76f2b076e14fcca79b8699d6be4704 (127.0.98.2:35207)
I20260927 14:57:19.182456 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:19.178133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55283 (local address 127.0.98.3:42813)
0927 14:57:19.178275 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:19.178279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:19.178294 (+ 15us) server_negotiation.cc:407] Connection header received
0927 14:57:19.178417 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:19.178420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:19.178467 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:19.178529 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:19.178867 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:19.179614 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:19.181149 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:19.181753 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:19.181849 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:19.181916 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:19.182234 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 14:57:19.182310 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 14:57:19.182456 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:19.178463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58231 (local address 127.0.98.2:35207)
0927 14:57:19.178604 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:19.178608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:19.178721 (+ 113us) server_negotiation.cc:407] Connection header received
0927 14:57:19.178852 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:19.178855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:19.178902 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:19.178975 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:19.179200 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:19.179842 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:19.181144 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:19.181742 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:19.181849 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:19.181916 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:19.182234 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 14:57:19.182310 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 14:57:19.182523 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:19.178045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42813 (local address 127.0.98.1:55283)
0927 14:57:19.178168 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:19.178184 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:19.178308 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:19.178584 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:19.178587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:19.178614 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:19.178739 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:19.178746 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:19.179722 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:19.179724 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:19.181024 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:19.181035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:19.181918 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:19.181922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:19.182107 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:19.182132 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 14:57:19.182214 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 14:57:19.182322 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":86}
I20260927 14:57:19.182538 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:19.178305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35207 (local address 127.0.98.1:58231)
0927 14:57:19.178647 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:19.178659 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:19.178752 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:19.178997 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:19.179000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:19.179017 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:19.179106 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:19.179110 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:19.179938 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:19.179941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:19.181035 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:19.181041 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:19.181918 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:19.181922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:19.182084 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:19.182129 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 14:57:19.182214 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 14:57:19.182330 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":36,"thread_start_us":131,"threads_started":1}
I20260927 14:57:19.183028 7550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "300a4a7704114716b98a41830284d499" candidate_uuid: "1544dca4984d47c494bb602d886ce11e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e76f2b076e14fcca79b8699d6be4704" is_pre_election: true
I20260927 14:57:19.183120 7687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "300a4a7704114716b98a41830284d499" candidate_uuid: "1544dca4984d47c494bb602d886ce11e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "280b4c9e2726475989c4e06525b89aec" is_pre_election: true
I20260927 14:57:19.183192 7550 raft_consensus.cc:2454] T 300a4a7704114716b98a41830284d499 P 5e76f2b076e14fcca79b8699d6be4704 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1544dca4984d47c494bb602d886ce11e in term 0.
I20260927 14:57:19.183367 7687 raft_consensus.cc:2454] T 300a4a7704114716b98a41830284d499 P 280b4c9e2726475989c4e06525b89aec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1544dca4984d47c494bb602d886ce11e in term 0.
I20260927 14:57:19.183583 7349 leader_election.cc:302] T 300a4a7704114716b98a41830284d499 P 1544dca4984d47c494bb602d886ce11e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1544dca4984d47c494bb602d886ce11e, 5e76f2b076e14fcca79b8699d6be4704; no voters:
I20260927 14:57:19.183849 7836 raft_consensus.cc:2781] T 300a4a7704114716b98a41830284d499 P 1544dca4984d47c494bb602d886ce11e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:57:19.183905 7836 raft_consensus.cc:482] T 300a4a7704114716b98a41830284d499 P 1544dca4984d47c494bb602d886ce11e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:57:19.183931 7836 raft_consensus.cc:3037] T 300a4a7704114716b98a41830284d499 P 1544dca4984d47c494bb602d886ce11e [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:19.184922 7836 raft_consensus.cc:504] T 300a4a7704114716b98a41830284d499 P 1544dca4984d47c494bb602d886ce11e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "280b4c9e2726475989c4e06525b89aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 } } peers { permanent_uuid: "1544dca4984d47c494bb602d886ce11e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "5e76f2b076e14fcca79b8699d6be4704" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35207 } }
I20260927 14:57:19.185262 7836 leader_election.cc:288] T 300a4a7704114716b98a41830284d499 P 1544dca4984d47c494bb602d886ce11e [CANDIDATE]: Term 1 election: Requested vote from peers 280b4c9e2726475989c4e06525b89aec (127.0.98.3:42813), 5e76f2b076e14fcca79b8699d6be4704 (127.0.98.2:35207)
I20260927 14:57:19.185741 7687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "300a4a7704114716b98a41830284d499" candidate_uuid: "1544dca4984d47c494bb602d886ce11e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "280b4c9e2726475989c4e06525b89aec"
I20260927 14:57:19.185741 7550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "300a4a7704114716b98a41830284d499" candidate_uuid: "1544dca4984d47c494bb602d886ce11e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e76f2b076e14fcca79b8699d6be4704"
I20260927 14:57:19.185883 7550 raft_consensus.cc:3037] T 300a4a7704114716b98a41830284d499 P 5e76f2b076e14fcca79b8699d6be4704 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:19.185886 7687 raft_consensus.cc:3037] T 300a4a7704114716b98a41830284d499 P 280b4c9e2726475989c4e06525b89aec [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:19.187105 7687 raft_consensus.cc:2454] T 300a4a7704114716b98a41830284d499 P 280b4c9e2726475989c4e06525b89aec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1544dca4984d47c494bb602d886ce11e in term 1.
I20260927 14:57:19.187105 7550 raft_consensus.cc:2454] T 300a4a7704114716b98a41830284d499 P 5e76f2b076e14fcca79b8699d6be4704 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1544dca4984d47c494bb602d886ce11e in term 1.
I20260927 14:57:19.187435 7349 leader_election.cc:302] T 300a4a7704114716b98a41830284d499 P 1544dca4984d47c494bb602d886ce11e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1544dca4984d47c494bb602d886ce11e, 280b4c9e2726475989c4e06525b89aec; no voters:
I20260927 14:57:19.187656 7836 raft_consensus.cc:2781] T 300a4a7704114716b98a41830284d499 P 1544dca4984d47c494bb602d886ce11e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:57:19.187878 7836 raft_consensus.cc:686] T 300a4a7704114716b98a41830284d499 P 1544dca4984d47c494bb602d886ce11e [term 1 LEADER]: Becoming Leader. State: Replica: 1544dca4984d47c494bb602d886ce11e, State: Running, Role: LEADER
I20260927 14:57:19.188148 7836 consensus_queue.cc:237] T 300a4a7704114716b98a41830284d499 P 1544dca4984d47c494bb602d886ce11e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "280b4c9e2726475989c4e06525b89aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 } } peers { permanent_uuid: "1544dca4984d47c494bb602d886ce11e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } } peers { permanent_uuid: "5e76f2b076e14fcca79b8699d6be4704" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35207 } }
I20260927 14:57:19.188617 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:19.188349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.98.1:51525)
0927 14:57:19.188472 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:19.188537 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 14:57:19.190614 7276 catalog_manager.cc:5495] T 300a4a7704114716b98a41830284d499 P 1544dca4984d47c494bb602d886ce11e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1544dca4984d47c494bb602d886ce11e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1544dca4984d47c494bb602d886ce11e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "280b4c9e2726475989c4e06525b89aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1544dca4984d47c494bb602d886ce11e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e76f2b076e14fcca79b8699d6be4704" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35207 } health_report { overall_health: UNKNOWN } } }
I20260927 14:57:19.204890 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:19.201437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40442 (local address 127.0.98.1:34875)
0927 14:57:19.201582 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:19.201586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:19.201599 (+ 13us) server_negotiation.cc:407] Connection header received
0927 14:57:19.201685 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:19.201688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:19.201745 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:19.201840 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:57:19.202089 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:19.202603 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:19.203470 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:19.203731 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:19.203821 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:19.204079 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:57:19.204574 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:57:19.204656 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:19.204714 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 14:57:19.204774 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 14:57:19.207856 7550 raft_consensus.cc:1261] T 300a4a7704114716b98a41830284d499 P 5e76f2b076e14fcca79b8699d6be4704 [term 1 FOLLOWER]: Refusing update from remote peer 1544dca4984d47c494bb602d886ce11e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:57:19.208081 7687 raft_consensus.cc:1261] T 300a4a7704114716b98a41830284d499 P 280b4c9e2726475989c4e06525b89aec [term 1 FOLLOWER]: Refusing update from remote peer 1544dca4984d47c494bb602d886ce11e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:57:19.208427 7836 consensus_queue.cc:1035] T 300a4a7704114716b98a41830284d499 P 1544dca4984d47c494bb602d886ce11e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e76f2b076e14fcca79b8699d6be4704" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:57:19.208616 7839 consensus_queue.cc:1035] T 300a4a7704114716b98a41830284d499 P 1544dca4984d47c494bb602d886ce11e [LEADER]: Connected to new peer: Peer: permanent_uuid: "280b4c9e2726475989c4e06525b89aec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:57:19.209600 7596 heartbeater.cc:503] Master 127.0.98.60:41351 was elected leader, sending a full tablet report...
I20260927 14:57:19.210301 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:19.210017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.98.2:60473)
0927 14:57:19.210169 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:19.210226 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 14:57:19.211792 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:19.211337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.98.3:48501)
0927 14:57:19.211659 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:19.211716 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257}
I20260927 14:57:19.212997 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:19.212767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.98.1:36991)
0927 14:57:19.212869 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:19.212924 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 14:57:19.214037 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:19.213794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.98.3:48339)
0927 14:57:19.213915 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:19.213969 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 14:57:19.214432 7845 mvcc.cc:205] Tried to move back new op lower bound from 7333974176589996032 to 7333974176517603328. Current Snapshot: MvccSnapshot[applied={T|T < 7333974176589996032}]
I20260927 14:57:19.217048 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:19.216806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.98.1:54213)
0927 14:57:19.216918 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:19.216974 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 14:57:19.220378 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:19.220027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.98.2:58517)
0927 14:57:19.220229 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:19.220290 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260927 14:57:19.224298 7276 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:57:19.224462 7276 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:57:19.228310 7276 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56ffb5d4c52e4a5fa29276ae2214dc99: Sending DeleteTablet for 3 replicas of tablet 300a4a7704114716b98a41830284d499
I20260927 14:57:19.229118 7393 tablet_service.cc:1514] Processing DeleteTablet for tablet 300a4a7704114716b98a41830284d499 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:57:19 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:40426
I20260927 14:57:19.229214 7667 tablet_service.cc:1514] Processing DeleteTablet for tablet 300a4a7704114716b98a41830284d499 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:57:19 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:60782
I20260927 14:57:19.229251 7530 tablet_service.cc:1514] Processing DeleteTablet for tablet 300a4a7704114716b98a41830284d499 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:57:19 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:42472
I20260927 14:57:19.229741 7853 tablet_replica.cc:330] T 300a4a7704114716b98a41830284d499 P 280b4c9e2726475989c4e06525b89aec: stopping tablet replica
I20260927 14:57:19.229871 7853 raft_consensus.cc:2229] T 300a4a7704114716b98a41830284d499 P 280b4c9e2726475989c4e06525b89aec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:57:19.229993 7853 raft_consensus.cc:2258] T 300a4a7704114716b98a41830284d499 P 280b4c9e2726475989c4e06525b89aec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:57:19.230230 7852 tablet_replica.cc:330] T 300a4a7704114716b98a41830284d499 P 5e76f2b076e14fcca79b8699d6be4704: stopping tablet replica
I20260927 14:57:19.230317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 7235
I20260927 14:57:19.230345 7852 raft_consensus.cc:2229] T 300a4a7704114716b98a41830284d499 P 5e76f2b076e14fcca79b8699d6be4704 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:57:19.230470 7852 raft_consensus.cc:2258] T 300a4a7704114716b98a41830284d499 P 5e76f2b076e14fcca79b8699d6be4704 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:57:19.231189 7852 ts_tablet_manager.cc:1905] T 300a4a7704114716b98a41830284d499 P 5e76f2b076e14fcca79b8699d6be4704: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:57:19.231233 7853 ts_tablet_manager.cc:1905] T 300a4a7704114716b98a41830284d499 P 280b4c9e2726475989c4e06525b89aec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:57:19.233309 7853 ts_tablet_manager.cc:1918] T 300a4a7704114716b98a41830284d499 P 280b4c9e2726475989c4e06525b89aec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:57:19.233385 7853 log.cc:1196] T 300a4a7704114716b98a41830284d499 P 280b4c9e2726475989c4e06525b89aec: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-2/wal/wals/300a4a7704114716b98a41830284d499
I20260927 14:57:19.233639 7853 ts_tablet_manager.cc:1939] T 300a4a7704114716b98a41830284d499 P 280b4c9e2726475989c4e06525b89aec: Deleting consensus metadata
I20260927 14:57:19.233731 7852 ts_tablet_manager.cc:1918] T 300a4a7704114716b98a41830284d499 P 5e76f2b076e14fcca79b8699d6be4704: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:57:19.233808 7852 log.cc:1196] T 300a4a7704114716b98a41830284d499 P 5e76f2b076e14fcca79b8699d6be4704: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790521004237705-392-0/minicluster-data/ts-1/wal/wals/300a4a7704114716b98a41830284d499
I20260927 14:57:19.234185 7852 ts_tablet_manager.cc:1939] T 300a4a7704114716b98a41830284d499 P 5e76f2b076e14fcca79b8699d6be4704: Deleting consensus metadata
I20260927 14:57:19.234405 7267 catalog_manager.cc:4841] TS 280b4c9e2726475989c4e06525b89aec (127.0.98.3:42813): tablet 300a4a7704114716b98a41830284d499 (table test-table [id=2f401691a8354232818a4a5087cb70fa]) successfully deleted
I20260927 14:57:19.234850 7267 catalog_manager.cc:4841] TS 5e76f2b076e14fcca79b8699d6be4704 (127.0.98.2:35207): tablet 300a4a7704114716b98a41830284d499 (table test-table [id=2f401691a8354232818a4a5087cb70fa]) successfully deleted
I20260927 14:57:19.239288 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 7467
I20260927 14:57:19.239429 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:19.239184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34875 (local address 127.0.0.1:40444)
0927 14:57:19.239292 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:19.239342 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 14:57:19.240088 7264 catalog_manager.cc:4584] TS 1544dca4984d47c494bb602d886ce11e (127.0.98.1:34875): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 300a4a7704114716b98a41830284d499: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34875: connect: Connection refused (error 111)
I20260927 14:57:19.244933 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 7603
I20260927 14:57:19.250895 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 6952
I20260927 14:57:19.256019 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 7061
2026-09-27T14:57:19Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5399 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:57:19 dist-test-slave-pz0h krb5kdc[7856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:57:19 dist-test-slave-pz0h krb5kdc[7856](info): set up 2 sockets
Sep 27 14:57:19 dist-test-slave-pz0h krb5kdc[7856](info): commencing operation
krb5kdc: starting...
W20260927 14:57:20.215169 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 14:57:20 dist-test-slave-pz0h krb5kdc[7856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:57:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:57:20Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 14:57:20.332993 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40335
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:20.445557 7869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:20.445852 7869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:20.445894 7869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:20.450132 7869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:57:20.450207 7869 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 14:57:20.450223 7869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:20.450240 7869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:20.450255 7869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:57:20.450273 7869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:57:20.455847 7869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40335
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:20.457019 7869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:20.458109 7869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:20.466753 7875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:20.466758 7874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:20.466753 7877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:20.466958 7869 server_base.cc:1034] running on GCE node
I20260927 14:57:20.467657 7869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:20.468848 7869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:20.470016 7869 hybrid_clock.cc:648] HybridClock initialized: now 1790521040469998 us; error 32 us; skew 500 ppm
Sep 27 14:57:20 dist-test-slave-pz0h krb5kdc[7856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521040, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:20.473398 7869 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 14:57:20.474974 7869 webserver.cc:460] Webserver started at http://127.0.98.62:36817/ using document root <none> and password file <none>
I20260927 14:57:20.475554 7869 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:20.475625 7869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:20.475816 7869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:20.477977 7869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "70a2a9a816574f60ba13f6996f2a64a7"
format_stamp: "Formatted at 2026-09-27 14:57:20 on dist-test-slave-pz0h"
server_key: "d680fba3b3e5188befcab85a0c1af440"
server_key_iv: "ec3d54688d1e93f542c857887af797a2"
server_key_version: "encryptionkey@0"
I20260927 14:57:20.478560 7869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "70a2a9a816574f60ba13f6996f2a64a7"
format_stamp: "Formatted at 2026-09-27 14:57:20 on dist-test-slave-pz0h"
server_key: "d680fba3b3e5188befcab85a0c1af440"
server_key_iv: "ec3d54688d1e93f542c857887af797a2"
server_key_version: "encryptionkey@0"
I20260927 14:57:20.482570 7869 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260927 14:57:20.485271 7884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:20.486563 7869 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:57:20.486707 7869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "70a2a9a816574f60ba13f6996f2a64a7"
format_stamp: "Formatted at 2026-09-27 14:57:20 on dist-test-slave-pz0h"
server_key: "d680fba3b3e5188befcab85a0c1af440"
server_key_iv: "ec3d54688d1e93f542c857887af797a2"
server_key_version: "encryptionkey@0"
I20260927 14:57:20.486857 7869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:20.503216 7869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:20.506446 7869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:20.506664 7869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:20.515277 7869 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40335
I20260927 14:57:20.515394 7946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40335 every 8 connection(s)
I20260927 14:57:20.516408 7869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:57:20.519524 7947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:20.519724 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 7869
I20260927 14:57:20.519842 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 14:57:20.520099 392 external_mini_cluster.cc:1428] Setting key fcaad18999cf32a1c5e092702630de6a
I20260927 14:57:20.526275 7947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70a2a9a816574f60ba13f6996f2a64a7: Bootstrap starting.
I20260927 14:57:20.529083 7947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70a2a9a816574f60ba13f6996f2a64a7: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:20.530045 7947 log.cc:824] T 00000000000000000000000000000000 P 70a2a9a816574f60ba13f6996f2a64a7: Log is configured to *not* fsync() on all Append() calls
Sep 27 14:57:20 dist-test-slave-pz0h krb5kdc[7856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 14:57:20.532501 7947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70a2a9a816574f60ba13f6996f2a64a7: No bootstrap required, opened a new log
I20260927 14:57:20.536521 7947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70a2a9a816574f60ba13f6996f2a64a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70a2a9a816574f60ba13f6996f2a64a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40335 } }
I20260927 14:57:20.536815 7947 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70a2a9a816574f60ba13f6996f2a64a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:20.536867 7947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70a2a9a816574f60ba13f6996f2a64a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70a2a9a816574f60ba13f6996f2a64a7, State: Initialized, Role: FOLLOWER
I20260927 14:57:20.537319 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:20.521512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52612 (local address 127.0.98.62:40335)
0927 14:57:20.522025 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:20.522040 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:20.523196 (+ 1156us) server_negotiation.cc:407] Connection header received
0927 14:57:20.523957 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:20.523977 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:20.524364 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:20.524695 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:20.525662 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:20.526754 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:20.527930 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:20.528404 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:20.528534 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:20.531298 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:20.531305 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:20.531323 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:20.531356 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:20.533940 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:20.534432 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:20.534439 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:20.534445 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:20.534594 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:20.534927 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:20.534931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:20.534933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:20.535497 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:20.535710 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:20.535991 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 14:57:20.536307 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":160,"threads_started":1}
I20260927 14:57:20.537354 7947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70a2a9a816574f60ba13f6996f2a64a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70a2a9a816574f60ba13f6996f2a64a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40335 } }
I20260927 14:57:20.537511 7947 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70a2a9a816574f60ba13f6996f2a64a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:57:20.537567 7947 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70a2a9a816574f60ba13f6996f2a64a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:57:20.537654 7947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70a2a9a816574f60ba13f6996f2a64a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:20.538621 7947 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70a2a9a816574f60ba13f6996f2a64a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70a2a9a816574f60ba13f6996f2a64a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40335 } }
I20260927 14:57:20.538985 7947 leader_election.cc:302] T 00000000000000000000000000000000 P 70a2a9a816574f60ba13f6996f2a64a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70a2a9a816574f60ba13f6996f2a64a7; no voters:
I20260927 14:57:20.539291 7947 leader_election.cc:288] T 00000000000000000000000000000000 P 70a2a9a816574f60ba13f6996f2a64a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:57:20.539410 7952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70a2a9a816574f60ba13f6996f2a64a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:57:20.539628 7952 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70a2a9a816574f60ba13f6996f2a64a7 [term 1 LEADER]: Becoming Leader. State: Replica: 70a2a9a816574f60ba13f6996f2a64a7, State: Running, Role: LEADER
I20260927 14:57:20.539942 7952 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70a2a9a816574f60ba13f6996f2a64a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70a2a9a816574f60ba13f6996f2a64a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40335 } }
I20260927 14:57:20.540498 7947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70a2a9a816574f60ba13f6996f2a64a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:57:20.541500 7953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70a2a9a816574f60ba13f6996f2a64a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70a2a9a816574f60ba13f6996f2a64a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70a2a9a816574f60ba13f6996f2a64a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40335 } } }
I20260927 14:57:20.541630 7953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70a2a9a816574f60ba13f6996f2a64a7 [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:20.541913 7954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70a2a9a816574f60ba13f6996f2a64a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70a2a9a816574f60ba13f6996f2a64a7. Latest consensus state: current_term: 1 leader_uuid: "70a2a9a816574f60ba13f6996f2a64a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70a2a9a816574f60ba13f6996f2a64a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40335 } } }
I20260927 14:57:20.542011 7954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70a2a9a816574f60ba13f6996f2a64a7 [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:20.542218 7961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:57:20.545428 7961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:57:20.551890 7961 catalog_manager.cc:1347] Generated new cluster ID: 11aa5fc65e5549298bcb0563e6d2a568
I20260927 14:57:20.551980 7961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:57:20.566421 7961 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:57:20.567808 7961 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:57:20.573201 7961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70a2a9a816574f60ba13f6996f2a64a7: Generated new TSK 0
I20260927 14:57:20.573915 7961 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 14:57:20.612046 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40335
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:20.745677 7973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:20.745986 7973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:20.746034 7973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:20.750612 7973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:20.750686 7973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:20.750732 7973 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 14:57:20.750787 7973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:57:20.756445 7973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:20.757692 7973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:20.758942 7973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:20.768925 7979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:20.768925 7981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:20.768965 7973 server_base.cc:1034] running on GCE node
W20260927 14:57:20.768967 7978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:20.769542 7973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:20.770249 7973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:20.771441 7973 hybrid_clock.cc:648] HybridClock initialized: now 1790521040771420 us; error 36 us; skew 500 ppm
Sep 27 14:57:20 dist-test-slave-pz0h krb5kdc[7856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521040, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:20.774240 7973 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 14:57:20.775220 7973 webserver.cc:460] Webserver started at http://127.0.98.1:41537/ using document root <none> and password file <none>
I20260927 14:57:20.775768 7973 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:20.775828 7973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:20.776003 7973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:20.778059 7973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-0/data/instance:
uuid: "f867c265ecbc44ec8494a1c145a40761"
format_stamp: "Formatted at 2026-09-27 14:57:20 on dist-test-slave-pz0h"
server_key: "c67ceea12bee77fa3530a58f1b958326"
server_key_iv: "90ac44ba5010ebd0164f9e68d9a406fe"
server_key_version: "encryptionkey@0"
I20260927 14:57:20.778613 7973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f867c265ecbc44ec8494a1c145a40761"
format_stamp: "Formatted at 2026-09-27 14:57:20 on dist-test-slave-pz0h"
server_key: "c67ceea12bee77fa3530a58f1b958326"
server_key_iv: "90ac44ba5010ebd0164f9e68d9a406fe"
server_key_version: "encryptionkey@0"
I20260927 14:57:20.782377 7973 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 14:57:20.785059 7988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:20.786360 7973 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 14:57:20.786484 7973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "f867c265ecbc44ec8494a1c145a40761"
format_stamp: "Formatted at 2026-09-27 14:57:20 on dist-test-slave-pz0h"
server_key: "c67ceea12bee77fa3530a58f1b958326"
server_key_iv: "90ac44ba5010ebd0164f9e68d9a406fe"
server_key_version: "encryptionkey@0"
I20260927 14:57:20.786612 7973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:20.801687 7973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:20.804971 7973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:20.805160 7973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:20.806710 7973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:20.806763 7973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:20.806833 7973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:20.806864 7973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 14:57:20 dist-test-slave-pz0h krb5kdc[7856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521040, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 14:57:20.817476 7973 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45889
I20260927 14:57:20.817533 8103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45889 every 8 connection(s)
I20260927 14:57:20.818411 7973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:57:20.819545 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 7973
I20260927 14:57:20.819669 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:57:20.819931 392 external_mini_cluster.cc:1428] Setting key ec56c48b01c45dd01f1a8fa531bfa90c
I20260927 14:57:20.820498 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:20.807581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50465 (local address 127.0.98.62:40335)
0927 14:57:20.807696 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:20.807700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:20.808780 (+ 1080us) server_negotiation.cc:407] Connection header received
0927 14:57:20.809779 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:20.809783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:20.809839 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:20.809916 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:20.811168 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:20.812007 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:20.813253 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:20.813552 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:20.813653 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:20.816621 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:20.816624 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:20.816626 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:20.816654 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:20.818203 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:20.818768 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:20.818772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:20.818773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:20.818900 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:20.819274 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:20.819293 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:20.819295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:20.819541 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:20.819654 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:20.820232 (+ 578us) server_negotiation.cc:299] Negotiation successful
0927 14:57:20.820361 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 14:57:20.821399 7997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:20.808135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40335 (local address 127.0.98.1:50465)
0927 14:57:20.808626 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:20.808669 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:20.809546 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:20.810100 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:20.810107 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:20.810461 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:20.811009 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:20.811026 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:20.812145 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:20.812149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:20.813118 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:20.813130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:20.813724 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:20.813729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:20.813910 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:20.814550 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:20.814573 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:20.816416 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:20.818332 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:20.818339 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:20.818355 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:20.818661 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:20.819021 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:20.819024 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:20.819026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:20.819186 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:20.819713 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:20.819720 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:20.819974 (+ 254us) client_negotiation.cc:769] Sending connection context
0927 14:57:20.820167 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 14:57:20.820493 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":138,"threads_started":1}
I20260927 14:57:20.822726 8104 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40335
I20260927 14:57:20.822953 8104 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:20.823558 8104 heartbeater.cc:511] Master 127.0.98.62:40335 requested a full tablet report, sending...
I20260927 14:57:20.825208 7900 ts_manager.cc:194] Registered new tserver with Master: f867c265ecbc44ec8494a1c145a40761 (127.0.98.1:45889)
I20260927 14:57:20.826323 7900 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50465
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260927 14:57:20.833478 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:20.824889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56449 (local address 127.0.98.62:40335)
0927 14:57:20.825018 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:20.825022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:20.827099 (+ 2077us) server_negotiation.cc:407] Connection header received
0927 14:57:20.827254 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:20.827258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:20.827300 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:20.827362 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:20.827783 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:20.828295 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:20.829080 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:20.829294 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:20.829378 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:20.830493 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:20.830496 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:20.830499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:20.830523 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:20.831967 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:20.832361 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:20.832365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:20.832367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:20.832443 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:20.832682 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:20.832685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:20.832687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:20.832871 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:20.832986 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:20.833209 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 14:57:20.833323 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 14:57:20.833505 8111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:20.824822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40335 (local address 127.0.98.1:56449)
0927 14:57:20.827001 (+ 2179us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:20.827016 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:20.827124 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:20.827392 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:20.827395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:20.827586 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:20.827683 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:20.827689 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:20.828408 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:20.828411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:20.828979 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:20.828987 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:20.829389 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:20.829393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:20.829486 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:20.829958 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:20.829970 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:20.830359 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:20.832074 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:20.832077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:20.832079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:20.832274 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:20.832524 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:20.832526 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:20.832528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:20.832603 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:20.832961 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:20.832963 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:20.833059 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 14:57:20.833147 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 14:57:20.833304 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2093,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 14:57:20.845896 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40335
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:20.958447 8113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:20.958763 8113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:20.958845 8113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:20.963107 8113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:20.963176 8113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:20.963217 8113 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 14:57:20.963254 8113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:57:20.968820 8113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:20.969940 8113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:20.971055 8113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:20.980681 8119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:20.980798 8113 server_base.cc:1034] running on GCE node
W20260927 14:57:20.980672 8118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:20.980675 8121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:20.981307 8113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:20.981945 8113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:20.983107 8113 hybrid_clock.cc:648] HybridClock initialized: now 1790521040983079 us; error 40 us; skew 500 ppm
Sep 27 14:57:20 dist-test-slave-pz0h krb5kdc[7856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521040, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:20.985919 8113 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 14:57:20.986913 8113 webserver.cc:460] Webserver started at http://127.0.98.2:40561/ using document root <none> and password file <none>
I20260927 14:57:20.987504 8113 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:20.987569 8113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:20.987763 8113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:20.989972 8113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-1/data/instance:
uuid: "021153c09e1748cda533bbdadf53d10a"
format_stamp: "Formatted at 2026-09-27 14:57:20 on dist-test-slave-pz0h"
server_key: "db486adc88e46d770a07e51bf8d8a003"
server_key_iv: "060356db82524fbbfeebcd1446423a94"
server_key_version: "encryptionkey@0"
I20260927 14:57:20.990577 8113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "021153c09e1748cda533bbdadf53d10a"
format_stamp: "Formatted at 2026-09-27 14:57:20 on dist-test-slave-pz0h"
server_key: "db486adc88e46d770a07e51bf8d8a003"
server_key_iv: "060356db82524fbbfeebcd1446423a94"
server_key_version: "encryptionkey@0"
I20260927 14:57:20.994524 8113 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260927 14:57:20.997148 8128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:20.998474 8113 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 14:57:20.998621 8113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "021153c09e1748cda533bbdadf53d10a"
format_stamp: "Formatted at 2026-09-27 14:57:20 on dist-test-slave-pz0h"
server_key: "db486adc88e46d770a07e51bf8d8a003"
server_key_iv: "060356db82524fbbfeebcd1446423a94"
server_key_version: "encryptionkey@0"
I20260927 14:57:20.998739 8113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:21.012547 8113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:21.015662 8113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:21.015853 8113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:21.017310 8113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:21.017364 8113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:21.017418 8113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:21.017447 8113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 14:57:21 dist-test-slave-pz0h krb5kdc[7856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521040, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 14:57:21.027683 8113 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46375
I20260927 14:57:21.027730 8243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46375 every 8 connection(s)
I20260927 14:57:21.028949 8113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:57:21.032032 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.018220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52761 (local address 127.0.98.62:40335)
0927 14:57:21.018341 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:21.018344 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:21.019551 (+ 1207us) server_negotiation.cc:407] Connection header received
0927 14:57:21.020689 (+ 1138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:21.020693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:21.020754 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:21.020830 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:21.022482 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.023264 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.024634 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.024858 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.024960 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.027605 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:21.027609 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:21.027612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:21.027645 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:21.029408 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:21.030043 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:21.030046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:21.030048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:21.030141 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:21.030665 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:21.030668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:21.030669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:21.031082 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:21.031185 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:21.031758 (+ 573us) server_negotiation.cc:299] Negotiation successful
0927 14:57:21.031891 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 14:57:21.032639 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 8113
I20260927 14:57:21.032753 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:57:21.032933 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.018772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40335 (local address 127.0.98.2:52761)
0927 14:57:21.019358 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:21.019403 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:21.020436 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:21.021073 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:21.021082 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:21.021568 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:21.022296 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.022318 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.023422 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.023426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.024495 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.024506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.025009 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.025013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.025195 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.025839 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:21.025863 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:21.027399 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:21.029560 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:21.029566 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:21.029582 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:21.029939 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:21.030227 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:21.030230 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:21.030231 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:21.030369 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:21.031218 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:21.031225 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:21.031501 (+ 276us) client_negotiation.cc:769] Sending connection context
0927 14:57:21.031689 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 14:57:21.031999 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":155,"threads_started":1}
I20260927 14:57:21.033041 392 external_mini_cluster.cc:1428] Setting key f16240f6a2ce475d202dcf31d2f28a29
I20260927 14:57:21.034421 8244 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40335
I20260927 14:57:21.034683 8244 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:21.035338 8244 heartbeater.cc:511] Master 127.0.98.62:40335 requested a full tablet report, sending...
I20260927 14:57:21.036944 7901 ts_manager.cc:194] Registered new tserver with Master: 021153c09e1748cda533bbdadf53d10a (127.0.98.2:46375)
I20260927 14:57:21.037588 7901 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52761
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 14:57:21.045152 8251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.037070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40335 (local address 127.0.98.2:51905)
0927 14:57:21.037352 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:21.037366 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:21.037497 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:21.037835 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:21.037838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:21.038060 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:21.038189 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.038196 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.039255 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.039259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.039852 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.039862 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.040464 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.040468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.040584 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.041197 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:21.041213 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:21.041635 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:21.043662 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:21.043665 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:21.043666 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:21.043945 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:21.044219 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:21.044221 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:21.044223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:21.044300 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:21.044661 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:21.044664 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:21.044774 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 14:57:21.044871 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 14:57:21.044999 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":98,"threads_started":1}
I20260927 14:57:21.045178 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.036941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51905 (local address 127.0.98.62:40335)
0927 14:57:21.037072 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:21.037077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:21.037431 (+ 354us) server_negotiation.cc:407] Connection header received
0927 14:57:21.037603 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:21.037607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:21.037654 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:21.037702 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:21.038314 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.039094 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.039987 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.040347 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.040465 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.041789 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:21.041794 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:21.041797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:21.041830 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:21.043535 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:21.044051 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:21.044054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:21.044055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:21.044145 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:21.044358 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:21.044361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:21.044362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:21.044576 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:21.044673 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:21.044942 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 14:57:21.045050 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 14:57:21.061708 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40335
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:21.174976 8253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:21.175273 8253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:21.175319 8253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:21.179514 8253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:21.179584 8253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:21.179625 8253 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 14:57:21.179664 8253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:57:21.185124 8253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:21.186223 8253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:21.187310 8253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:21.196924 8259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:21.196992 8253 server_base.cc:1034] running on GCE node
W20260927 14:57:21.196924 8258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:21.196928 8261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:21.197546 8253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:21.198190 8253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:21.199333 8253 hybrid_clock.cc:648] HybridClock initialized: now 1790521041199293 us; error 54 us; skew 500 ppm
Sep 27 14:57:21 dist-test-slave-pz0h krb5kdc[7856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521041, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:21.202135 8253 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 14:57:21.203220 8253 webserver.cc:460] Webserver started at http://127.0.98.3:41065/ using document root <none> and password file <none>
I20260927 14:57:21.203889 8253 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:21.203976 8253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:21.204187 8253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:21.206576 8253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-2/data/instance:
uuid: "fd7728cbbc3c44329af3426868d6aa91"
format_stamp: "Formatted at 2026-09-27 14:57:21 on dist-test-slave-pz0h"
server_key: "e86d2d6d04d3fff4c07caeb508976758"
server_key_iv: "73de5bf7c5c84e78248412ee2863fba9"
server_key_version: "encryptionkey@0"
I20260927 14:57:21.207274 8253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fd7728cbbc3c44329af3426868d6aa91"
format_stamp: "Formatted at 2026-09-27 14:57:21 on dist-test-slave-pz0h"
server_key: "e86d2d6d04d3fff4c07caeb508976758"
server_key_iv: "73de5bf7c5c84e78248412ee2863fba9"
server_key_version: "encryptionkey@0"
I20260927 14:57:21.211469 8253 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 14:57:21.214177 8268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:21.215538 8253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:57:21.215672 8253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "fd7728cbbc3c44329af3426868d6aa91"
format_stamp: "Formatted at 2026-09-27 14:57:21 on dist-test-slave-pz0h"
server_key: "e86d2d6d04d3fff4c07caeb508976758"
server_key_iv: "73de5bf7c5c84e78248412ee2863fba9"
server_key_version: "encryptionkey@0"
I20260927 14:57:21.215790 8253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:21.247625 8253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:21.250543 8253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:21.250727 8253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:21.252218 8253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:21.252275 8253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:21.252331 8253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:21.252358 8253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 14:57:21 dist-test-slave-pz0h krb5kdc[7856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521041, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 14:57:21.262228 8253 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40285
I20260927 14:57:21.262256 8383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40285 every 8 connection(s)
I20260927 14:57:21.263255 8253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:57:21.265659 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.253082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39389 (local address 127.0.98.62:40335)
0927 14:57:21.253197 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:21.253200 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:21.254316 (+ 1116us) server_negotiation.cc:407] Connection header received
0927 14:57:21.255367 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:21.255371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:21.255432 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:21.255503 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:21.256972 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.257797 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.259398 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.259627 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.259719 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.262498 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:21.262501 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:21.262503 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:21.262530 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:21.263936 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:21.264436 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:21.264439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:21.264440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:21.264520 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:21.264773 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:21.264775 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:21.264776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:21.264943 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:21.265014 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:21.265434 (+ 420us) server_negotiation.cc:299] Negotiation successful
0927 14:57:21.265531 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 14:57:21.266419 8277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.253638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40335 (local address 127.0.98.3:39389)
0927 14:57:21.254155 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:21.254199 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:21.255131 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:21.255691 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:21.255700 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:21.256171 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:21.256808 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.256829 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.257953 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.257957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.259284 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.259296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.259773 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.259777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.259951 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.260605 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:21.260628 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:21.262318 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:21.264055 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:21.264061 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:21.264075 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:21.264348 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:21.264586 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:21.264588 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:21.264589 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:21.264713 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:21.265017 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:21.265021 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:21.265251 (+ 230us) client_negotiation.cc:769] Sending connection context
0927 14:57:21.265402 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 14:57:21.265646 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":137,"threads_started":1}
I20260927 14:57:21.267545 8384 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40335
I20260927 14:57:21.267696 8384 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:21.268172 8384 heartbeater.cc:511] Master 127.0.98.62:40335 requested a full tablet report, sending...
I20260927 14:57:21.268532 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 8253
I20260927 14:57:21.268651 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:57:21.268965 392 external_mini_cluster.cc:1428] Setting key c24707472ef9d5deea56849f22bd4d72
I20260927 14:57:21.269205 7900 ts_manager.cc:194] Registered new tserver with Master: fd7728cbbc3c44329af3426868d6aa91 (127.0.98.3:40285)
I20260927 14:57:21.269840 7900 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39389
I20260927 14:57:21.271093 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:57:21.277645 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.270350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54107 (local address 127.0.98.62:40335)
0927 14:57:21.270458 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:21.270463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:21.270693 (+ 230us) server_negotiation.cc:407] Connection header received
0927 14:57:21.270876 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:21.270880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:21.270927 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:21.270991 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:21.271486 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.272195 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.273066 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.273289 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.273365 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.274468 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:21.274472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:21.274475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:21.274501 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:21.276070 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:21.276516 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:21.276519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:21.276520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:21.276593 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:21.276849 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:21.276851 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:21.276852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:21.277061 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:21.277143 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:21.277393 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 14:57:21.277519 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 14:57:21.277715 8390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.270291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40335 (local address 127.0.98.3:54107)
0927 14:57:21.270614 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:21.270628 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:21.270761 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:21.271028 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:21.271031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:21.271257 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:21.271367 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.271375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.272322 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.272325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.272952 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.272962 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.273389 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.273392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.273483 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.273967 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:21.273979 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:21.274329 (+ 350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:21.276173 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:21.276177 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:21.276180 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:21.276424 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:21.276672 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:21.276675 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:21.276677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:21.276770 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:21.277140 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:21.277143 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:21.277242 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 14:57:21.277338 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 14:57:21.277509 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":96,"threads_started":1}
Sep 27 14:57:21 dist-test-slave-pz0h krb5kdc[7856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521041, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 14:57:21 dist-test-slave-pz0h krb5kdc[7856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:21 dist-test-slave-pz0h krb5kdc[7856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 14:57:21.295800 7950 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:21.289432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52626 (local address 127.0.98.62:40335)
0927 14:57:21.289574 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:21.289578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:21.289592 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:57:21.289634 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:21.289637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:21.289681 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:21.289777 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:21.290486 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.291106 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.292012 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.292237 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.292329 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.295662 (+ 3333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52626: BlockingRecv error: recv got EOF from 127.0.0.1:52626 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 14:57:21 dist-test-slave-pz0h krb5kdc[7856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521041, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 14:57:21.305223 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.297208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52642 (local address 127.0.98.62:40335)
0927 14:57:21.297332 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:21.297337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:21.297411 (+ 74us) server_negotiation.cc:407] Connection header received
0927 14:57:21.297550 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:21.297552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:21.297593 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:21.297658 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:21.298246 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.298757 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.299579 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.299828 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.299920 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.302310 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:21.302313 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:21.302315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:21.302345 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:21.303655 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:21.304058 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:21.304062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:21.304064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:21.304141 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:21.304381 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:21.304384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:21.304386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:21.304589 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:21.304711 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:21.304931 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 14:57:21.305064 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:57:21.308344 7900 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:57:21.311327 7900 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:57:21.326634 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.321032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51048 (local address 127.0.98.1:45889)
0927 14:57:21.322294 (+ 1262us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:21.322310 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:21.322332 (+ 22us) server_negotiation.cc:407] Connection header received
0927 14:57:21.322401 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:21.322406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:21.322611 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:21.322766 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:21.323015 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.323909 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.325203 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.325817 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.325927 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.326096 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:21.326381 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 14:57:21.326489 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1153,"thread_start_us":78,"threads_started":1}
I20260927 14:57:21.326640 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.320292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54980 (local address 127.0.98.2:46375)
0927 14:57:21.321306 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:21.321319 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:21.321343 (+ 24us) server_negotiation.cc:407] Connection header received
0927 14:57:21.321419 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:21.321424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:21.321662 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:21.321824 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:21.322267 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.323434 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.325291 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.326083 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.326180 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.326339 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:21.326401 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 14:57:21.326500 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":907,"thread_start_us":102,"threads_started":1}
I20260927 14:57:21.326627 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.320134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46375 (local address 127.0.0.1:54980)
0927 14:57:21.320524 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:21.320574 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:21.320700 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:21.321872 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:21.321879 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:21.321905 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:21.322053 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.322059 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.323542 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.323545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.325177 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.325186 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.326177 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.326180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.326261 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.326281 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 14:57:21.326329 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 14:57:21.326418 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":544,"thread_start_us":79,"threads_started":1}
I20260927 14:57:21.326632 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.320193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40285 (local address 127.0.0.1:38396)
0927 14:57:21.320954 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:21.320967 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:21.321062 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:21.321991 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:21.321995 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:21.322013 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:21.322114 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.322121 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.323359 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.323362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.324915 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.324927 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.326053 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.326057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.326152 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.326178 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 14:57:21.326225 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 14:57:21.326415 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"mutex_wait_us":11,"thread_start_us":136,"threads_started":1}
I20260927 14:57:21.327685 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.321118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38396 (local address 127.0.98.3:40285)
0927 14:57:21.321597 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:21.321607 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:21.321629 (+ 22us) server_negotiation.cc:407] Connection header received
0927 14:57:21.321696 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:21.321701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:21.321868 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:21.322022 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:21.322239 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.323262 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.325054 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.325882 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.326720 (+ 838us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.327391 (+ 671us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:21.327444 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 14:57:21.327543 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":94,"threads_started":1}
I20260927 14:57:21.328508 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.320849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45889 (local address 127.0.0.1:51048)
0927 14:57:21.321365 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:21.321379 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:21.321473 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:21.322743 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:21.322746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:21.322770 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:21.322887 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.322893 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.324007 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.324010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.325075 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.325085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.325945 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.325949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.326088 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.326133 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 14:57:21.326244 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 14:57:21.328409 (+ 2165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":2108,"thread_start_us":79,"threads_started":1}
I20260927 14:57:21.330098 8318 tablet_service.cc:1467] Processing CreateTablet for tablet 65989ddc75b54239a3447bc71207c67e (DEFAULT_TABLE table=test-table [id=05b1a327befb4ee0ad5af321f5166acd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:57:21.330000 8178 tablet_service.cc:1467] Processing CreateTablet for tablet 65989ddc75b54239a3447bc71207c67e (DEFAULT_TABLE table=test-table [id=05b1a327befb4ee0ad5af321f5166acd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:57:21.330755 8038 tablet_service.cc:1467] Processing CreateTablet for tablet 65989ddc75b54239a3447bc71207c67e (DEFAULT_TABLE table=test-table [id=05b1a327befb4ee0ad5af321f5166acd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:57:21.331216 8178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65989ddc75b54239a3447bc71207c67e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:21.331224 8318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65989ddc75b54239a3447bc71207c67e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:21.331823 8038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65989ddc75b54239a3447bc71207c67e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:21.338027 8412 tablet_bootstrap.cc:492] T 65989ddc75b54239a3447bc71207c67e P f867c265ecbc44ec8494a1c145a40761: Bootstrap starting.
I20260927 14:57:21.340013 8413 tablet_bootstrap.cc:492] T 65989ddc75b54239a3447bc71207c67e P fd7728cbbc3c44329af3426868d6aa91: Bootstrap starting.
I20260927 14:57:21.340529 8412 tablet_bootstrap.cc:654] T 65989ddc75b54239a3447bc71207c67e P f867c265ecbc44ec8494a1c145a40761: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:21.340538 8414 tablet_bootstrap.cc:492] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a: Bootstrap starting.
I20260927 14:57:21.341274 8412 log.cc:824] T 65989ddc75b54239a3447bc71207c67e P f867c265ecbc44ec8494a1c145a40761: Log is configured to *not* fsync() on all Append() calls
I20260927 14:57:21.343194 8413 tablet_bootstrap.cc:654] T 65989ddc75b54239a3447bc71207c67e P fd7728cbbc3c44329af3426868d6aa91: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:21.343261 8412 tablet_bootstrap.cc:492] T 65989ddc75b54239a3447bc71207c67e P f867c265ecbc44ec8494a1c145a40761: No bootstrap required, opened a new log
I20260927 14:57:21.343446 8412 ts_tablet_manager.cc:1397] T 65989ddc75b54239a3447bc71207c67e P f867c265ecbc44ec8494a1c145a40761: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 14:57:21.344039 8414 tablet_bootstrap.cc:654] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:21.344033 8413 log.cc:824] T 65989ddc75b54239a3447bc71207c67e P fd7728cbbc3c44329af3426868d6aa91: Log is configured to *not* fsync() on all Append() calls
I20260927 14:57:21.344820 8414 log.cc:824] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a: Log is configured to *not* fsync() on all Append() calls
I20260927 14:57:21.346144 8413 tablet_bootstrap.cc:492] T 65989ddc75b54239a3447bc71207c67e P fd7728cbbc3c44329af3426868d6aa91: No bootstrap required, opened a new log
I20260927 14:57:21.346336 8413 ts_tablet_manager.cc:1397] T 65989ddc75b54239a3447bc71207c67e P fd7728cbbc3c44329af3426868d6aa91: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260927 14:57:21.346735 8414 tablet_bootstrap.cc:492] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a: No bootstrap required, opened a new log
I20260927 14:57:21.346948 8414 ts_tablet_manager.cc:1397] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 14:57:21.347215 8412 raft_consensus.cc:348] T 65989ddc75b54239a3447bc71207c67e P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } } peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } }
I20260927 14:57:21.347434 8412 raft_consensus.cc:374] T 65989ddc75b54239a3447bc71207c67e P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:21.347493 8412 raft_consensus.cc:729] T 65989ddc75b54239a3447bc71207c67e P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f867c265ecbc44ec8494a1c145a40761, State: Initialized, Role: FOLLOWER
I20260927 14:57:21.347910 8412 consensus_queue.cc:260] T 65989ddc75b54239a3447bc71207c67e P f867c265ecbc44ec8494a1c145a40761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } } peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } }
I20260927 14:57:21.348699 8412 ts_tablet_manager.cc:1428] T 65989ddc75b54239a3447bc71207c67e P f867c265ecbc44ec8494a1c145a40761: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 14:57:21.349057 8104 heartbeater.cc:503] Master 127.0.98.62:40335 was elected leader, sending a full tablet report...
I20260927 14:57:21.350495 8413 raft_consensus.cc:348] T 65989ddc75b54239a3447bc71207c67e P fd7728cbbc3c44329af3426868d6aa91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } } peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } }
I20260927 14:57:21.350495 8414 raft_consensus.cc:348] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } } peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } }
I20260927 14:57:21.350762 8413 raft_consensus.cc:374] T 65989ddc75b54239a3447bc71207c67e P fd7728cbbc3c44329af3426868d6aa91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:21.350762 8414 raft_consensus.cc:374] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:21.350842 8414 raft_consensus.cc:729] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021153c09e1748cda533bbdadf53d10a, State: Initialized, Role: FOLLOWER
I20260927 14:57:21.350842 8413 raft_consensus.cc:729] T 65989ddc75b54239a3447bc71207c67e P fd7728cbbc3c44329af3426868d6aa91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd7728cbbc3c44329af3426868d6aa91, State: Initialized, Role: FOLLOWER
I20260927 14:57:21.351296 8413 consensus_queue.cc:260] T 65989ddc75b54239a3447bc71207c67e P fd7728cbbc3c44329af3426868d6aa91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } } peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } }
I20260927 14:57:21.351296 8414 consensus_queue.cc:260] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } } peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } }
I20260927 14:57:21.352172 8384 heartbeater.cc:503] Master 127.0.98.62:40335 was elected leader, sending a full tablet report...
I20260927 14:57:21.352214 8244 heartbeater.cc:503] Master 127.0.98.62:40335 was elected leader, sending a full tablet report...
I20260927 14:57:21.352325 8414 ts_tablet_manager.cc:1428] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 14:57:21.353423 8413 ts_tablet_manager.cc:1428] T 65989ddc75b54239a3447bc71207c67e P fd7728cbbc3c44329af3426868d6aa91: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 14:57:21.367532 8420 raft_consensus.cc:482] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:57:21.367786 8420 raft_consensus.cc:504] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } } peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } }
I20260927 14:57:21.369089 8420 leader_election.cc:288] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd7728cbbc3c44329af3426868d6aa91 (127.0.98.3:40285), f867c265ecbc44ec8494a1c145a40761 (127.0.98.1:45889)
I20260927 14:57:21.373217 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.369394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52461 (local address 127.0.98.1:45889)
0927 14:57:21.369550 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:21.369555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:21.369568 (+ 13us) server_negotiation.cc:407] Connection header received
0927 14:57:21.369622 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:21.369625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:21.369675 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:21.369764 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:21.370043 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.370862 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.372112 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.372725 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.372808 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.372864 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:21.373054 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 14:57:21.373111 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 14:57:21.373260 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.369288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45889 (local address 127.0.98.2:52461)
0927 14:57:21.369391 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:21.369405 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:21.369490 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:21.369774 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:21.369776 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:21.369797 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:21.369931 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.369937 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.370959 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.370962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.371987 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.371996 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.372836 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.372840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.372942 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.372968 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 14:57:21.373040 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 14:57:21.373117 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"spinlock_wait_cycles":1792}
I20260927 14:57:21.373895 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.369418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42363 (local address 127.0.98.3:40285)
0927 14:57:21.369528 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:21.369532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:21.369813 (+ 281us) server_negotiation.cc:407] Connection header received
0927 14:57:21.369904 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:21.369907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:21.369956 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:21.370022 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:21.370778 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.371543 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.372741 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.373306 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.373400 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.373459 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:21.373738 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 14:57:21.373805 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 14:57:21.373909 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.369288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40285 (local address 127.0.98.2:42363)
0927 14:57:21.369722 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:21.369736 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:21.369823 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:21.370038 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:21.370040 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:21.370057 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:21.370662 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.370673 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.371649 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.371651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.372608 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.372618 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.373499 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.373503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.373604 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.373624 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 14:57:21.373697 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 14:57:21.373766 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":108,"thread_start_us":138,"threads_started":1}
I20260927 14:57:21.373987 8058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65989ddc75b54239a3447bc71207c67e" candidate_uuid: "021153c09e1748cda533bbdadf53d10a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f867c265ecbc44ec8494a1c145a40761" is_pre_election: true
I20260927 14:57:21.374315 8058 raft_consensus.cc:2454] T 65989ddc75b54239a3447bc71207c67e P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021153c09e1748cda533bbdadf53d10a in term 0.
I20260927 14:57:21.374558 8338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65989ddc75b54239a3447bc71207c67e" candidate_uuid: "021153c09e1748cda533bbdadf53d10a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd7728cbbc3c44329af3426868d6aa91" is_pre_election: true
I20260927 14:57:21.374830 8132 leader_election.cc:302] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021153c09e1748cda533bbdadf53d10a, f867c265ecbc44ec8494a1c145a40761; no voters:
I20260927 14:57:21.374863 8338 raft_consensus.cc:2454] T 65989ddc75b54239a3447bc71207c67e P fd7728cbbc3c44329af3426868d6aa91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021153c09e1748cda533bbdadf53d10a in term 0.
I20260927 14:57:21.375079 8420 raft_consensus.cc:2781] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:57:21.375159 8420 raft_consensus.cc:482] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:57:21.375212 8420 raft_consensus.cc:3037] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:21.376214 8420 raft_consensus.cc:504] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } } peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } }
I20260927 14:57:21.376545 8420 leader_election.cc:288] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a [CANDIDATE]: Term 1 election: Requested vote from peers fd7728cbbc3c44329af3426868d6aa91 (127.0.98.3:40285), f867c265ecbc44ec8494a1c145a40761 (127.0.98.1:45889)
I20260927 14:57:21.376866 8338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65989ddc75b54239a3447bc71207c67e" candidate_uuid: "021153c09e1748cda533bbdadf53d10a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd7728cbbc3c44329af3426868d6aa91"
I20260927 14:57:21.376883 8058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65989ddc75b54239a3447bc71207c67e" candidate_uuid: "021153c09e1748cda533bbdadf53d10a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f867c265ecbc44ec8494a1c145a40761"
I20260927 14:57:21.376981 8338 raft_consensus.cc:3037] T 65989ddc75b54239a3447bc71207c67e P fd7728cbbc3c44329af3426868d6aa91 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:21.376987 8058 raft_consensus.cc:3037] T 65989ddc75b54239a3447bc71207c67e P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:21.378012 8058 raft_consensus.cc:2454] T 65989ddc75b54239a3447bc71207c67e P f867c265ecbc44ec8494a1c145a40761 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021153c09e1748cda533bbdadf53d10a in term 1.
I20260927 14:57:21.378103 8338 raft_consensus.cc:2454] T 65989ddc75b54239a3447bc71207c67e P fd7728cbbc3c44329af3426868d6aa91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021153c09e1748cda533bbdadf53d10a in term 1.
I20260927 14:57:21.378322 8132 leader_election.cc:302] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021153c09e1748cda533bbdadf53d10a, f867c265ecbc44ec8494a1c145a40761; no voters:
I20260927 14:57:21.378500 8420 raft_consensus.cc:2781] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:57:21.378717 8420 raft_consensus.cc:686] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a [term 1 LEADER]: Becoming Leader. State: Replica: 021153c09e1748cda533bbdadf53d10a, State: Running, Role: LEADER
I20260927 14:57:21.379050 8420 consensus_queue.cc:237] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } } peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } }
I20260927 14:57:21.382182 7901 catalog_manager.cc:5495] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a reported cstate change: term changed from 0 to 1, leader changed from <none> to 021153c09e1748cda533bbdadf53d10a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "021153c09e1748cda533bbdadf53d10a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } health_report { overall_health: UNKNOWN } } }
I20260927 14:57:21.404874 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.401528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54984 (local address 127.0.98.2:46375)
0927 14:57:21.401656 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:21.401660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:21.401673 (+ 13us) server_negotiation.cc:407] Connection header received
0927 14:57:21.401746 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:21.401748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:21.401804 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:21.401899 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:57:21.402096 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.402680 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.403562 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.403801 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.403889 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.404111 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:57:21.404549 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:57:21.404637 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:21.404682 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 14:57:21.404739 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 14:57:21.410715 8338 raft_consensus.cc:1261] T 65989ddc75b54239a3447bc71207c67e P fd7728cbbc3c44329af3426868d6aa91 [term 1 FOLLOWER]: Refusing update from remote peer 021153c09e1748cda533bbdadf53d10a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:57:21.410715 8058 raft_consensus.cc:1261] T 65989ddc75b54239a3447bc71207c67e P f867c265ecbc44ec8494a1c145a40761 [term 1 FOLLOWER]: Refusing update from remote peer 021153c09e1748cda533bbdadf53d10a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:57:21.411518 8420 consensus_queue.cc:1035] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:57:21.411741 8422 consensus_queue.cc:1035] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:57:21.419152 8425 mvcc.cc:205] Tried to move back new op lower bound from 7333974185610948608 to 7333974185491025920. Current Snapshot: MvccSnapshot[applied={T|T < 7333974185610948608}]
I20260927 14:57:21.420807 8426 mvcc.cc:205] Tried to move back new op lower bound from 7333974185610948608 to 7333974185491025920. Current Snapshot: MvccSnapshot[applied={T|T < 7333974185610948608}]
I20260927 14:57:21.429148 7899 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:57:21.429346 7899 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:57:21.431808 7899 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70a2a9a816574f60ba13f6996f2a64a7: Sending DeleteTablet for 3 replicas of tablet 65989ddc75b54239a3447bc71207c67e
I20260927 14:57:21.432633 8318 tablet_service.cc:1514] Processing DeleteTablet for tablet 65989ddc75b54239a3447bc71207c67e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:57:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38396
I20260927 14:57:21.432811 8178 tablet_service.cc:1514] Processing DeleteTablet for tablet 65989ddc75b54239a3447bc71207c67e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:57:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54980
I20260927 14:57:21.433169 8435 tablet_replica.cc:330] T 65989ddc75b54239a3447bc71207c67e P fd7728cbbc3c44329af3426868d6aa91: stopping tablet replica
I20260927 14:57:21.433276 8038 tablet_service.cc:1514] Processing DeleteTablet for tablet 65989ddc75b54239a3447bc71207c67e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:57:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51048
I20260927 14:57:21.433228 7899 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 14:57:21.433496 8435 raft_consensus.cc:2229] T 65989ddc75b54239a3447bc71207c67e P fd7728cbbc3c44329af3426868d6aa91 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 14:57:21.433657 7899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:57:21.433818 8435 raft_consensus.cc:2258] T 65989ddc75b54239a3447bc71207c67e P fd7728cbbc3c44329af3426868d6aa91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:57:21.434114 8436 tablet_replica.cc:330] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a: stopping tablet replica
I20260927 14:57:21.434442 8437 tablet_replica.cc:330] T 65989ddc75b54239a3447bc71207c67e P f867c265ecbc44ec8494a1c145a40761: stopping tablet replica
I20260927 14:57:21.435109 8437 raft_consensus.cc:2229] T 65989ddc75b54239a3447bc71207c67e P f867c265ecbc44ec8494a1c145a40761 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:57:21.435223 8436 raft_consensus.cc:2229] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:57:21.435545 8435 ts_tablet_manager.cc:1905] T 65989ddc75b54239a3447bc71207c67e P fd7728cbbc3c44329af3426868d6aa91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:57:21.435858 8437 raft_consensus.cc:2258] T 65989ddc75b54239a3447bc71207c67e P f867c265ecbc44ec8494a1c145a40761 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:57:21.436223 8436 raft_consensus.cc:2258] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:57:21.438295 8437 ts_tablet_manager.cc:1905] T 65989ddc75b54239a3447bc71207c67e P f867c265ecbc44ec8494a1c145a40761: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:57:21.438653 8435 ts_tablet_manager.cc:1918] T 65989ddc75b54239a3447bc71207c67e P fd7728cbbc3c44329af3426868d6aa91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:57:21.438761 8435 log.cc:1196] T 65989ddc75b54239a3447bc71207c67e P fd7728cbbc3c44329af3426868d6aa91: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-2/wal/wals/65989ddc75b54239a3447bc71207c67e
I20260927 14:57:21.439288 8318 tablet_service.cc:1467] Processing CreateTablet for tablet 43403cf4524e4bc1876e8e0afd7ac4c3 (DEFAULT_TABLE table=test-table [id=191598badae344f2b3ff9903675744b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:57:21.439431 8038 tablet_service.cc:1467] Processing CreateTablet for tablet 43403cf4524e4bc1876e8e0afd7ac4c3 (DEFAULT_TABLE table=test-table [id=191598badae344f2b3ff9903675744b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:57:21.439550 8318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43403cf4524e4bc1876e8e0afd7ac4c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:21.439683 8038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43403cf4524e4bc1876e8e0afd7ac4c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:21.441475 8437 ts_tablet_manager.cc:1918] T 65989ddc75b54239a3447bc71207c67e P f867c265ecbc44ec8494a1c145a40761: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:57:21.441666 8437 log.cc:1196] T 65989ddc75b54239a3447bc71207c67e P f867c265ecbc44ec8494a1c145a40761: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-0/wal/wals/65989ddc75b54239a3447bc71207c67e
I20260927 14:57:21.441818 8413 tablet_bootstrap.cc:492] T 43403cf4524e4bc1876e8e0afd7ac4c3 P fd7728cbbc3c44329af3426868d6aa91: Bootstrap starting.
I20260927 14:57:21.442016 8412 tablet_bootstrap.cc:492] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761: Bootstrap starting.
I20260927 14:57:21.442306 8178 tablet_service.cc:1467] Processing CreateTablet for tablet 43403cf4524e4bc1876e8e0afd7ac4c3 (DEFAULT_TABLE table=test-table [id=191598badae344f2b3ff9903675744b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:57:21.442469 8435 ts_tablet_manager.cc:1939] T 65989ddc75b54239a3447bc71207c67e P fd7728cbbc3c44329af3426868d6aa91: Deleting consensus metadata
I20260927 14:57:21.442613 8178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43403cf4524e4bc1876e8e0afd7ac4c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:21.442900 8413 tablet_bootstrap.cc:654] T 43403cf4524e4bc1876e8e0afd7ac4c3 P fd7728cbbc3c44329af3426868d6aa91: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:21.443079 8412 tablet_bootstrap.cc:654] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:21.443805 8437 ts_tablet_manager.cc:1939] T 65989ddc75b54239a3447bc71207c67e P f867c265ecbc44ec8494a1c145a40761: Deleting consensus metadata
I20260927 14:57:21.443852 7887 catalog_manager.cc:4841] TS fd7728cbbc3c44329af3426868d6aa91 (127.0.98.3:40285): tablet 65989ddc75b54239a3447bc71207c67e (table test-table [id=05b1a327befb4ee0ad5af321f5166acd]) successfully deleted
I20260927 14:57:21.444829 8413 tablet_bootstrap.cc:492] T 43403cf4524e4bc1876e8e0afd7ac4c3 P fd7728cbbc3c44329af3426868d6aa91: No bootstrap required, opened a new log
I20260927 14:57:21.444916 8413 ts_tablet_manager.cc:1397] T 43403cf4524e4bc1876e8e0afd7ac4c3 P fd7728cbbc3c44329af3426868d6aa91: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 14:57:21.445115 8414 tablet_bootstrap.cc:492] T 43403cf4524e4bc1876e8e0afd7ac4c3 P 021153c09e1748cda533bbdadf53d10a: Bootstrap starting.
I20260927 14:57:21.445356 8413 raft_consensus.cc:348] T 43403cf4524e4bc1876e8e0afd7ac4c3 P fd7728cbbc3c44329af3426868d6aa91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:21.445474 8413 raft_consensus.cc:374] T 43403cf4524e4bc1876e8e0afd7ac4c3 P fd7728cbbc3c44329af3426868d6aa91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:21.445510 8413 raft_consensus.cc:729] T 43403cf4524e4bc1876e8e0afd7ac4c3 P fd7728cbbc3c44329af3426868d6aa91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd7728cbbc3c44329af3426868d6aa91, State: Initialized, Role: FOLLOWER
I20260927 14:57:21.445638 8413 consensus_queue.cc:260] T 43403cf4524e4bc1876e8e0afd7ac4c3 P fd7728cbbc3c44329af3426868d6aa91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:21.445894 8413 ts_tablet_manager.cc:1428] T 43403cf4524e4bc1876e8e0afd7ac4c3 P fd7728cbbc3c44329af3426868d6aa91: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 14:57:21.445972 8412 tablet_bootstrap.cc:492] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761: No bootstrap required, opened a new log
I20260927 14:57:21.446069 8414 tablet_bootstrap.cc:654] T 43403cf4524e4bc1876e8e0afd7ac4c3 P 021153c09e1748cda533bbdadf53d10a: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:21.446066 8412 ts_tablet_manager.cc:1397] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 14:57:21.446543 8436 ts_tablet_manager.cc:1905] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:57:21.446672 8412 raft_consensus.cc:348] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:21.446776 8412 raft_consensus.cc:374] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:21.446830 8412 raft_consensus.cc:729] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f867c265ecbc44ec8494a1c145a40761, State: Initialized, Role: FOLLOWER
I20260927 14:57:21.446909 7888 catalog_manager.cc:4841] TS f867c265ecbc44ec8494a1c145a40761 (127.0.98.1:45889): tablet 65989ddc75b54239a3447bc71207c67e (table test-table [id=05b1a327befb4ee0ad5af321f5166acd]) successfully deleted
I20260927 14:57:21.446949 8412 consensus_queue.cc:260] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:21.447190 8412 ts_tablet_manager.cc:1428] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:57:21.449003 8436 ts_tablet_manager.cc:1918] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:57:21.449110 8436 log.cc:1196] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-1/wal/wals/65989ddc75b54239a3447bc71207c67e
I20260927 14:57:21.449388 8436 ts_tablet_manager.cc:1939] T 65989ddc75b54239a3447bc71207c67e P 021153c09e1748cda533bbdadf53d10a: Deleting consensus metadata
I20260927 14:57:21.450533 7888 catalog_manager.cc:4841] TS 021153c09e1748cda533bbdadf53d10a (127.0.98.2:46375): tablet 65989ddc75b54239a3447bc71207c67e (table test-table [id=05b1a327befb4ee0ad5af321f5166acd]) successfully deleted
I20260927 14:57:21.451097 8414 tablet_bootstrap.cc:492] T 43403cf4524e4bc1876e8e0afd7ac4c3 P 021153c09e1748cda533bbdadf53d10a: No bootstrap required, opened a new log
I20260927 14:57:21.451176 8414 ts_tablet_manager.cc:1397] T 43403cf4524e4bc1876e8e0afd7ac4c3 P 021153c09e1748cda533bbdadf53d10a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 14:57:21.451582 8414 raft_consensus.cc:348] T 43403cf4524e4bc1876e8e0afd7ac4c3 P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:21.451684 8414 raft_consensus.cc:374] T 43403cf4524e4bc1876e8e0afd7ac4c3 P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:21.451714 8414 raft_consensus.cc:729] T 43403cf4524e4bc1876e8e0afd7ac4c3 P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021153c09e1748cda533bbdadf53d10a, State: Initialized, Role: FOLLOWER
I20260927 14:57:21.451826 8414 consensus_queue.cc:260] T 43403cf4524e4bc1876e8e0afd7ac4c3 P 021153c09e1748cda533bbdadf53d10a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:21.452090 8414 ts_tablet_manager.cc:1428] T 43403cf4524e4bc1876e8e0afd7ac4c3 P 021153c09e1748cda533bbdadf53d10a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:57:21.479537 8418 raft_consensus.cc:482] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:57:21.479676 8418 raft_consensus.cc:504] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:21.480908 8418 leader_election.cc:288] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 021153c09e1748cda533bbdadf53d10a (127.0.98.2:46375), fd7728cbbc3c44329af3426868d6aa91 (127.0.98.3:40285)
I20260927 14:57:21.485194 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.481287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44683 (local address 127.0.98.3:40285)
0927 14:57:21.481408 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:21.481413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:21.481762 (+ 349us) server_negotiation.cc:407] Connection header received
0927 14:57:21.481909 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:21.481913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:21.481958 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:21.482001 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:21.482382 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.483108 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.484126 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.484630 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.484729 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.484793 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:21.484991 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 14:57:21.485071 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 14:57:21.485209 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.481118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40285 (local address 127.0.98.1:44683)
0927 14:57:21.481668 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:21.481681 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:21.481789 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:21.482130 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:21.482133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:21.482148 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:21.482276 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.482281 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.483206 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.483209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.484015 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.484023 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.484757 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.484761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.484855 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.484880 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 14:57:21.484962 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 14:57:21.485032 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":213,"thread_start_us":104,"threads_started":1}
I20260927 14:57:21.485569 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.481175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57009 (local address 127.0.98.2:46375)
0927 14:57:21.481310 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:21.481314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:21.481537 (+ 223us) server_negotiation.cc:407] Connection header received
0927 14:57:21.481718 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:21.481720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:21.481775 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:21.481838 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:21.482113 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.482869 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.484348 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.484913 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.485006 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.485069 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:21.485357 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 14:57:21.485437 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:57:21.485909 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.481110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46375 (local address 127.0.98.1:57009)
0927 14:57:21.481455 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:21.481469 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:21.481611 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:21.481868 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:21.481870 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:21.481886 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:21.482012 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.482017 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.482982 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.482985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.484222 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.484232 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.485125 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.485129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.485218 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.485240 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 14:57:21.485690 (+ 450us) client_negotiation.cc:240] Negotiation successful
0927 14:57:21.485761 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":79,"threads_started":1}
I20260927 14:57:21.485984 8338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43403cf4524e4bc1876e8e0afd7ac4c3" candidate_uuid: "f867c265ecbc44ec8494a1c145a40761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd7728cbbc3c44329af3426868d6aa91" is_pre_election: true
I20260927 14:57:21.486148 8338 raft_consensus.cc:2454] T 43403cf4524e4bc1876e8e0afd7ac4c3 P fd7728cbbc3c44329af3426868d6aa91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f867c265ecbc44ec8494a1c145a40761 in term 0.
I20260927 14:57:21.486469 7991 leader_election.cc:302] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f867c265ecbc44ec8494a1c145a40761, fd7728cbbc3c44329af3426868d6aa91; no voters:
I20260927 14:57:21.486498 8198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43403cf4524e4bc1876e8e0afd7ac4c3" candidate_uuid: "f867c265ecbc44ec8494a1c145a40761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021153c09e1748cda533bbdadf53d10a" is_pre_election: true
I20260927 14:57:21.486704 8198 raft_consensus.cc:2454] T 43403cf4524e4bc1876e8e0afd7ac4c3 P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f867c265ecbc44ec8494a1c145a40761 in term 0.
I20260927 14:57:21.486742 8418 raft_consensus.cc:2781] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:57:21.486814 8418 raft_consensus.cc:482] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:57:21.486857 8418 raft_consensus.cc:3037] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:21.487870 8418 raft_consensus.cc:504] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:21.488220 8418 leader_election.cc:288] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761 [CANDIDATE]: Term 1 election: Requested vote from peers 021153c09e1748cda533bbdadf53d10a (127.0.98.2:46375), fd7728cbbc3c44329af3426868d6aa91 (127.0.98.3:40285)
I20260927 14:57:21.488570 8338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43403cf4524e4bc1876e8e0afd7ac4c3" candidate_uuid: "f867c265ecbc44ec8494a1c145a40761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd7728cbbc3c44329af3426868d6aa91"
I20260927 14:57:21.488699 8338 raft_consensus.cc:3037] T 43403cf4524e4bc1876e8e0afd7ac4c3 P fd7728cbbc3c44329af3426868d6aa91 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:21.489832 8338 raft_consensus.cc:2454] T 43403cf4524e4bc1876e8e0afd7ac4c3 P fd7728cbbc3c44329af3426868d6aa91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f867c265ecbc44ec8494a1c145a40761 in term 1.
I20260927 14:57:21.490140 7991 leader_election.cc:302] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f867c265ecbc44ec8494a1c145a40761, fd7728cbbc3c44329af3426868d6aa91; no voters:
I20260927 14:57:21.490226 8198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43403cf4524e4bc1876e8e0afd7ac4c3" candidate_uuid: "f867c265ecbc44ec8494a1c145a40761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021153c09e1748cda533bbdadf53d10a"
I20260927 14:57:21.490337 8418 raft_consensus.cc:2781] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:57:21.490362 8198 raft_consensus.cc:3037] T 43403cf4524e4bc1876e8e0afd7ac4c3 P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:21.490556 8418 raft_consensus.cc:686] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761 [term 1 LEADER]: Becoming Leader. State: Replica: f867c265ecbc44ec8494a1c145a40761, State: Running, Role: LEADER
I20260927 14:57:21.490851 8418 consensus_queue.cc:237] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:21.492195 8198 raft_consensus.cc:2454] T 43403cf4524e4bc1876e8e0afd7ac4c3 P 021153c09e1748cda533bbdadf53d10a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f867c265ecbc44ec8494a1c145a40761 in term 1.
I20260927 14:57:21.493448 7899 catalog_manager.cc:5495] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761 reported cstate change: term changed from 0 to 1, leader changed from <none> to f867c265ecbc44ec8494a1c145a40761 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f867c265ecbc44ec8494a1c145a40761" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } health_report { overall_health: UNKNOWN } } }
W20260927 14:57:21.514223 8385 tablet.cc:2369] T 43403cf4524e4bc1876e8e0afd7ac4c3 P fd7728cbbc3c44329af3426868d6aa91: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 14:57:21 dist-test-slave-pz0h krb5kdc[7856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521040, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 14:57:21.529349 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.521107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40335 (local address 127.0.0.1:52656)
0927 14:57:21.521378 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:21.521391 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:21.521515 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:21.521734 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:21.521737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:21.522189 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:21.522290 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.522298 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.523191 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.523194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.523791 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.523801 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.524227 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.524230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.524308 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.524848 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:21.524860 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:21.526236 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:21.527837 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:21.527846 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:21.527853 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:21.528091 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:21.528330 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:21.528333 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:21.528335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:21.528410 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:21.528760 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:21.528767 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:21.528872 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 14:57:21.528974 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 14:57:21.529163 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":1,"thread_start_us":117,"threads_started":1}
I20260927 14:57:21.529348 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.521119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52656 (local address 127.0.98.62:40335)
0927 14:57:21.521237 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:21.521242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:21.521460 (+ 218us) server_negotiation.cc:407] Connection header received
0927 14:57:21.521599 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:21.521601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:21.521640 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:21.521703 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:21.522404 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.523056 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.523892 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.524125 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.524211 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.526374 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:21.526377 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:21.526378 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:21.526400 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:21.527730 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:21.528179 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:21.528183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:21.528184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:21.528251 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:21.528483 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:21.528487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:21.528488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:21.528665 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:21.528749 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:21.529027 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 14:57:21.529228 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":45}
W20260927 14:57:21.530112 8245 tablet.cc:2369] T 43403cf4524e4bc1876e8e0afd7ac4c3 P 021153c09e1748cda533bbdadf53d10a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:57:21.533555 7899 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52656:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 14:57:21.534374 7899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:57:21.539335 8038 tablet_service.cc:1467] Processing CreateTablet for tablet 193dfa5e288346ee8f66665c809e7ce3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b5ba3877446241ecafb552fca601749e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 14:57:21.539528 8178 tablet_service.cc:1467] Processing CreateTablet for tablet 193dfa5e288346ee8f66665c809e7ce3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b5ba3877446241ecafb552fca601749e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 14:57:21.539528 8318 tablet_service.cc:1467] Processing CreateTablet for tablet 193dfa5e288346ee8f66665c809e7ce3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b5ba3877446241ecafb552fca601749e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 14:57:21.539764 8038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 193dfa5e288346ee8f66665c809e7ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:21.539887 8178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 193dfa5e288346ee8f66665c809e7ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:21.539902 8318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 193dfa5e288346ee8f66665c809e7ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:21.543193 8414 tablet_bootstrap.cc:492] T 193dfa5e288346ee8f66665c809e7ce3 P 021153c09e1748cda533bbdadf53d10a: Bootstrap starting.
I20260927 14:57:21.543265 8412 tablet_bootstrap.cc:492] T 193dfa5e288346ee8f66665c809e7ce3 P f867c265ecbc44ec8494a1c145a40761: Bootstrap starting.
I20260927 14:57:21.543457 8413 tablet_bootstrap.cc:492] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91: Bootstrap starting.
I20260927 14:57:21.544270 8414 tablet_bootstrap.cc:654] T 193dfa5e288346ee8f66665c809e7ce3 P 021153c09e1748cda533bbdadf53d10a: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:21.544269 8412 tablet_bootstrap.cc:654] T 193dfa5e288346ee8f66665c809e7ce3 P f867c265ecbc44ec8494a1c145a40761: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:21.544341 8413 tablet_bootstrap.cc:654] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:21.546021 8412 tablet_bootstrap.cc:492] T 193dfa5e288346ee8f66665c809e7ce3 P f867c265ecbc44ec8494a1c145a40761: No bootstrap required, opened a new log
I20260927 14:57:21.546135 8412 ts_tablet_manager.cc:1397] T 193dfa5e288346ee8f66665c809e7ce3 P f867c265ecbc44ec8494a1c145a40761: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 14:57:21.546468 8414 tablet_bootstrap.cc:492] T 193dfa5e288346ee8f66665c809e7ce3 P 021153c09e1748cda533bbdadf53d10a: No bootstrap required, opened a new log
I20260927 14:57:21.546551 8414 ts_tablet_manager.cc:1397] T 193dfa5e288346ee8f66665c809e7ce3 P 021153c09e1748cda533bbdadf53d10a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 14:57:21.546604 8412 raft_consensus.cc:348] T 193dfa5e288346ee8f66665c809e7ce3 P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:21.546747 8412 raft_consensus.cc:374] T 193dfa5e288346ee8f66665c809e7ce3 P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:21.546784 8412 raft_consensus.cc:729] T 193dfa5e288346ee8f66665c809e7ce3 P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f867c265ecbc44ec8494a1c145a40761, State: Initialized, Role: FOLLOWER
I20260927 14:57:21.546939 8412 consensus_queue.cc:260] T 193dfa5e288346ee8f66665c809e7ce3 P f867c265ecbc44ec8494a1c145a40761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:21.547021 8414 raft_consensus.cc:348] T 193dfa5e288346ee8f66665c809e7ce3 P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:21.547138 8414 raft_consensus.cc:374] T 193dfa5e288346ee8f66665c809e7ce3 P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:21.547168 8414 raft_consensus.cc:729] T 193dfa5e288346ee8f66665c809e7ce3 P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021153c09e1748cda533bbdadf53d10a, State: Initialized, Role: FOLLOWER
I20260927 14:57:21.547219 8412 ts_tablet_manager.cc:1428] T 193dfa5e288346ee8f66665c809e7ce3 P f867c265ecbc44ec8494a1c145a40761: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 14:57:21.547281 8414 consensus_queue.cc:260] T 193dfa5e288346ee8f66665c809e7ce3 P 021153c09e1748cda533bbdadf53d10a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:21.547412 8418 tablet_replica.cc:439] T 193dfa5e288346ee8f66665c809e7ce3 P f867c265ecbc44ec8494a1c145a40761: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } } }
I20260927 14:57:21.547523 8414 ts_tablet_manager.cc:1428] T 193dfa5e288346ee8f66665c809e7ce3 P 021153c09e1748cda533bbdadf53d10a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:57:21.547549 8418 tablet_replica.cc:442] T 193dfa5e288346ee8f66665c809e7ce3 P f867c265ecbc44ec8494a1c145a40761: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:57:21.547761 8432 tablet_replica.cc:439] T 193dfa5e288346ee8f66665c809e7ce3 P 021153c09e1748cda533bbdadf53d10a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } } }
I20260927 14:57:21.548022 8432 tablet_replica.cc:442] T 193dfa5e288346ee8f66665c809e7ce3 P 021153c09e1748cda533bbdadf53d10a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:57:21.549288 8413 tablet_bootstrap.cc:492] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91: No bootstrap required, opened a new log
I20260927 14:57:21.549383 8413 ts_tablet_manager.cc:1397] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260927 14:57:21.549790 8413 raft_consensus.cc:348] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:21.549906 8413 raft_consensus.cc:374] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:21.549953 8413 raft_consensus.cc:729] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd7728cbbc3c44329af3426868d6aa91, State: Initialized, Role: FOLLOWER
I20260927 14:57:21.550073 8413 consensus_queue.cc:260] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:21.550344 8413 ts_tablet_manager.cc:1428] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:57:21.550331 8419 tablet_replica.cc:439] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } } }
I20260927 14:57:21.550493 8419 tablet_replica.cc:442] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 14:57:21.569828 8105 tablet.cc:2369] T 193dfa5e288346ee8f66665c809e7ce3 P f867c265ecbc44ec8494a1c145a40761: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:57:21.739985 8419 raft_consensus.cc:482] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:57:21.740142 8419 raft_consensus.cc:504] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:21.741597 8419 leader_election.cc:288] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f867c265ecbc44ec8494a1c145a40761 (127.0.98.1:45889), 021153c09e1748cda533bbdadf53d10a (127.0.98.2:46375)
I20260927 14:57:21.745226 8277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.741896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45889 (local address 127.0.98.3:36855)
0927 14:57:21.742023 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:21.742041 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:21.742165 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:21.742489 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:21.742492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:21.742512 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:21.742619 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.742625 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.743364 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.743367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.744194 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.744203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.744801 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.744805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.744903 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.744922 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 14:57:21.744993 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 14:57:21.745069 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":22}
I20260927 14:57:21.745213 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.741976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36855 (local address 127.0.98.1:45889)
0927 14:57:21.742114 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:21.742119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:21.742136 (+ 17us) server_negotiation.cc:407] Connection header received
0927 14:57:21.742299 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:21.742303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:21.742360 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:21.742467 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:21.742729 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.743263 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.744310 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.744698 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.744786 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.744850 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:21.745017 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 14:57:21.745089 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 14:57:21.745726 8058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "193dfa5e288346ee8f66665c809e7ce3" candidate_uuid: "fd7728cbbc3c44329af3426868d6aa91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f867c265ecbc44ec8494a1c145a40761" is_pre_election: true
I20260927 14:57:21.745847 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.742161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46375 (local address 127.0.98.3:38333)
0927 14:57:21.742486 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:21.742499 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:21.742597 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:21.742826 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:21.742829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:21.742845 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:21.742959 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.742964 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.743631 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.743634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.744765 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.744773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.745462 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.745466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.745562 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.745583 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 14:57:21.745653 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 14:57:21.745715 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":3,"thread_start_us":74,"threads_started":1}
I20260927 14:57:21.745882 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.742310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38333 (local address 127.0.98.2:46375)
0927 14:57:21.742454 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:21.742458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:21.742582 (+ 124us) server_negotiation.cc:407] Connection header received
0927 14:57:21.742682 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:21.742685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:21.742729 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:21.742811 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:21.743058 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.743533 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.744884 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.745308 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.745408 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.745475 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:21.745695 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 14:57:21.745777 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 14:57:21.745878 8058 raft_consensus.cc:2454] T 193dfa5e288346ee8f66665c809e7ce3 P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd7728cbbc3c44329af3426868d6aa91 in term 0.
I20260927 14:57:21.746241 8272 leader_election.cc:302] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f867c265ecbc44ec8494a1c145a40761, fd7728cbbc3c44329af3426868d6aa91; no voters:
I20260927 14:57:21.746263 8198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "193dfa5e288346ee8f66665c809e7ce3" candidate_uuid: "fd7728cbbc3c44329af3426868d6aa91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021153c09e1748cda533bbdadf53d10a" is_pre_election: true
I20260927 14:57:21.746389 8198 raft_consensus.cc:2454] T 193dfa5e288346ee8f66665c809e7ce3 P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd7728cbbc3c44329af3426868d6aa91 in term 0.
I20260927 14:57:21.746500 8419 raft_consensus.cc:2781] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:57:21.746563 8419 raft_consensus.cc:482] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:57:21.746598 8419 raft_consensus.cc:3037] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:21.747628 8419 raft_consensus.cc:504] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:21.747941 8419 leader_election.cc:288] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91 [CANDIDATE]: Term 1 election: Requested vote from peers f867c265ecbc44ec8494a1c145a40761 (127.0.98.1:45889), 021153c09e1748cda533bbdadf53d10a (127.0.98.2:46375)
I20260927 14:57:21.748339 8058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "193dfa5e288346ee8f66665c809e7ce3" candidate_uuid: "fd7728cbbc3c44329af3426868d6aa91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f867c265ecbc44ec8494a1c145a40761"
I20260927 14:57:21.748339 8198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "193dfa5e288346ee8f66665c809e7ce3" candidate_uuid: "fd7728cbbc3c44329af3426868d6aa91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021153c09e1748cda533bbdadf53d10a"
I20260927 14:57:21.748466 8198 raft_consensus.cc:3037] T 193dfa5e288346ee8f66665c809e7ce3 P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:21.748466 8058 raft_consensus.cc:3037] T 193dfa5e288346ee8f66665c809e7ce3 P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:21.749565 8198 raft_consensus.cc:2454] T 193dfa5e288346ee8f66665c809e7ce3 P 021153c09e1748cda533bbdadf53d10a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd7728cbbc3c44329af3426868d6aa91 in term 1.
I20260927 14:57:21.749565 8058 raft_consensus.cc:2454] T 193dfa5e288346ee8f66665c809e7ce3 P f867c265ecbc44ec8494a1c145a40761 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd7728cbbc3c44329af3426868d6aa91 in term 1.
I20260927 14:57:21.749869 8272 leader_election.cc:302] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021153c09e1748cda533bbdadf53d10a, fd7728cbbc3c44329af3426868d6aa91; no voters:
I20260927 14:57:21.750098 8419 raft_consensus.cc:2781] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:57:21.750308 8419 raft_consensus.cc:686] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91 [term 1 LEADER]: Becoming Leader. State: Replica: fd7728cbbc3c44329af3426868d6aa91, State: Running, Role: LEADER
I20260927 14:57:21.750550 8419 consensus_queue.cc:237] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:21.751760 8455 tablet_replica.cc:439] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91: TxnStatusTablet state changed. Reason: New leader fd7728cbbc3c44329af3426868d6aa91. Latest consensus state: current_term: 1 leader_uuid: "fd7728cbbc3c44329af3426868d6aa91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } } }
I20260927 14:57:21.751883 8455 tablet_replica.cc:442] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91: This TxnStatusTablet replica's current role is: LEADER
I20260927 14:57:21.752154 8457 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 14:57:21.752300 8457 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 14:57:21.753053 7901 catalog_manager.cc:5495] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd7728cbbc3c44329af3426868d6aa91 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "fd7728cbbc3c44329af3426868d6aa91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } health_report { overall_health: HEALTHY } } }
I20260927 14:57:21.789364 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.785784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38404 (local address 127.0.98.3:40285)
0927 14:57:21.785912 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:21.785917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:21.786028 (+ 111us) server_negotiation.cc:407] Connection header received
0927 14:57:21.786223 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:21.786226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:21.786280 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:21.786356 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:57:21.786601 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.787143 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.787964 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.788177 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.788261 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.788550 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:57:21.788978 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:57:21.789087 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:21.789165 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 14:57:21.789219 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 14:57:21.789364 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.785697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40285 (local address 127.0.0.1:38404)
0927 14:57:21.785956 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:21.785971 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:21.786089 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:21.786379 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:21.786382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:21.786392 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 14:57:21.786506 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.786511 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.787244 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.787248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.787857 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.787865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.788274 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.788278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.788372 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.788418 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 14:57:21.789074 (+ 656us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 14:57:21.789091 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 14:57:21.789155 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 14:57:21.789248 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260927 14:57:21.792364 8058 raft_consensus.cc:1261] T 193dfa5e288346ee8f66665c809e7ce3 P f867c265ecbc44ec8494a1c145a40761 [term 1 FOLLOWER]: Refusing update from remote peer fd7728cbbc3c44329af3426868d6aa91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:57:21.792639 8198 raft_consensus.cc:1261] T 193dfa5e288346ee8f66665c809e7ce3 P 021153c09e1748cda533bbdadf53d10a [term 1 FOLLOWER]: Refusing update from remote peer fd7728cbbc3c44329af3426868d6aa91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:57:21.792920 8419 consensus_queue.cc:1035] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:57:21.793128 8455 consensus_queue.cc:1035] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:57:21.794996 8432 tablet_replica.cc:439] T 193dfa5e288346ee8f66665c809e7ce3 P 021153c09e1748cda533bbdadf53d10a: TxnStatusTablet state changed. Reason: New leader fd7728cbbc3c44329af3426868d6aa91. Latest consensus state: current_term: 1 leader_uuid: "fd7728cbbc3c44329af3426868d6aa91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } } }
I20260927 14:57:21.795137 8432 tablet_replica.cc:442] T 193dfa5e288346ee8f66665c809e7ce3 P 021153c09e1748cda533bbdadf53d10a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:57:21.795634 8418 tablet_replica.cc:439] T 193dfa5e288346ee8f66665c809e7ce3 P f867c265ecbc44ec8494a1c145a40761: TxnStatusTablet state changed. Reason: New leader fd7728cbbc3c44329af3426868d6aa91. Latest consensus state: current_term: 1 leader_uuid: "fd7728cbbc3c44329af3426868d6aa91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } } }
I20260927 14:57:21.795766 8418 tablet_replica.cc:442] T 193dfa5e288346ee8f66665c809e7ce3 P f867c265ecbc44ec8494a1c145a40761: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:57:21.796543 8427 mvcc.cc:205] Tried to move back new op lower bound from 7333974187176677376 to 7333974187012927488. Current Snapshot: MvccSnapshot[applied={T|T < 7333974187176677376}]
I20260927 14:57:21.796653 8455 tablet_replica.cc:439] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd7728cbbc3c44329af3426868d6aa91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } } }
I20260927 14:57:21.796741 8455 tablet_replica.cc:442] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91: This TxnStatusTablet replica's current role is: LEADER
I20260927 14:57:21.797748 8418 tablet_replica.cc:439] T 193dfa5e288346ee8f66665c809e7ce3 P f867c265ecbc44ec8494a1c145a40761: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd7728cbbc3c44329af3426868d6aa91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } } }
I20260927 14:57:21.797847 8418 tablet_replica.cc:442] T 193dfa5e288346ee8f66665c809e7ce3 P f867c265ecbc44ec8494a1c145a40761: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:57:21.798671 8455 tablet_replica.cc:439] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd7728cbbc3c44329af3426868d6aa91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } } }
I20260927 14:57:21.798774 8455 tablet_replica.cc:442] T 193dfa5e288346ee8f66665c809e7ce3 P fd7728cbbc3c44329af3426868d6aa91: This TxnStatusTablet replica's current role is: LEADER
I20260927 14:57:21.800038 8432 tablet_replica.cc:439] T 193dfa5e288346ee8f66665c809e7ce3 P 021153c09e1748cda533bbdadf53d10a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd7728cbbc3c44329af3426868d6aa91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } } }
I20260927 14:57:21.800156 8432 tablet_replica.cc:442] T 193dfa5e288346ee8f66665c809e7ce3 P 021153c09e1748cda533bbdadf53d10a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 14:57:21.806691 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.802868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51058 (local address 127.0.98.1:45889)
0927 14:57:21.802992 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:21.802996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:21.803042 (+ 46us) server_negotiation.cc:407] Connection header received
0927 14:57:21.803224 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:21.803228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:21.803290 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:21.803379 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:57:21.803659 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.804201 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.805325 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.805534 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.805600 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.805845 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:57:21.806296 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:57:21.806382 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:21.806462 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 14:57:21.806517 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 14:57:21.818022 8464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.814468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40285 (local address 127.0.98.1:37135)
0927 14:57:21.814784 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:21.814838 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:21.814956 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:21.815182 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:21.815185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:21.815194 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 14:57:21.815313 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.815318 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.816030 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.816034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.816661 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.816669 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.817242 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.817246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.817370 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.817426 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 14:57:21.817726 (+ 300us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 14:57:21.817733 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 14:57:21.817791 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 14:57:21.817878 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":109,"threads_started":1}
I20260927 14:57:21.818007 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.814592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37135 (local address 127.0.98.3:40285)
0927 14:57:21.814726 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:21.814730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:21.814902 (+ 172us) server_negotiation.cc:407] Connection header received
0927 14:57:21.815051 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:21.815054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:21.815101 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:21.815163 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:57:21.815406 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.815919 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.816771 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.817104 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.817218 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.817568 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:57:21.817651 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:57:21.817710 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:21.817801 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 14:57:21.817853 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 14:57:21.828480 8338 raft_consensus.cc:1261] T 43403cf4524e4bc1876e8e0afd7ac4c3 P fd7728cbbc3c44329af3426868d6aa91 [term 1 FOLLOWER]: Refusing update from remote peer f867c265ecbc44ec8494a1c145a40761: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:57:21.828543 8198 raft_consensus.cc:1261] T 43403cf4524e4bc1876e8e0afd7ac4c3 P 021153c09e1748cda533bbdadf53d10a [term 1 FOLLOWER]: Refusing update from remote peer f867c265ecbc44ec8494a1c145a40761: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:57:21.829005 8418 consensus_queue.cc:1035] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:57:21.829458 8443 consensus_queue.cc:1035] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:57:21.858172 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.853592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55081 (local address 127.0.98.1:45889)
0927 14:57:21.853716 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:21.853721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:21.854010 (+ 289us) server_negotiation.cc:407] Connection header received
0927 14:57:21.854152 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:21.854156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:21.854211 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:21.854275 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:57:21.854662 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.855496 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.856787 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.857133 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.857213 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.857655 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:57:21.857758 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:57:21.857803 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:21.857962 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 14:57:21.858028 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 14:57:21.858632 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.853461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45889 (local address 127.0.98.3:55081)
0927 14:57:21.853911 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:21.853927 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:21.854051 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:21.854395 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:21.854398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:21.854408 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 14:57:21.854533 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.854539 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.855649 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.855652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.856637 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:21.856649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.857335 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.857339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:21.857427 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.857495 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 14:57:21.857862 (+ 367us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 14:57:21.857868 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 14:57:21.858344 (+ 476us) client_negotiation.cc:240] Negotiation successful
0927 14:57:21.858442 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":108,"threads_started":1}
I20260927 14:57:21.886607 7899 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:57:21.886780 7899 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:57:21.888957 7899 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70a2a9a816574f60ba13f6996f2a64a7: Sending DeleteTablet for 3 replicas of tablet 43403cf4524e4bc1876e8e0afd7ac4c3
I20260927 14:57:21.889709 8178 tablet_service.cc:1514] Processing DeleteTablet for tablet 43403cf4524e4bc1876e8e0afd7ac4c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:57:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54980
I20260927 14:57:21.889777 8318 tablet_service.cc:1514] Processing DeleteTablet for tablet 43403cf4524e4bc1876e8e0afd7ac4c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:57:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38396
I20260927 14:57:21.889978 8435 tablet_replica.cc:330] T 43403cf4524e4bc1876e8e0afd7ac4c3 P fd7728cbbc3c44329af3426868d6aa91: stopping tablet replica
I20260927 14:57:21.890142 8435 raft_consensus.cc:2229] T 43403cf4524e4bc1876e8e0afd7ac4c3 P fd7728cbbc3c44329af3426868d6aa91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:57:21.890313 8435 raft_consensus.cc:2258] T 43403cf4524e4bc1876e8e0afd7ac4c3 P fd7728cbbc3c44329af3426868d6aa91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:57:21.890542 8038 tablet_service.cc:1514] Processing DeleteTablet for tablet 43403cf4524e4bc1876e8e0afd7ac4c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:57:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51048
I20260927 14:57:21.890781 8437 tablet_replica.cc:330] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761: stopping tablet replica
I20260927 14:57:21.890919 8437 raft_consensus.cc:2229] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761 [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:57:21.891280 8437 raft_consensus.cc:2258] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:57:21.891659 8436 tablet_replica.cc:330] T 43403cf4524e4bc1876e8e0afd7ac4c3 P 021153c09e1748cda533bbdadf53d10a: stopping tablet replica
I20260927 14:57:21.891805 8436 raft_consensus.cc:2229] T 43403cf4524e4bc1876e8e0afd7ac4c3 P 021153c09e1748cda533bbdadf53d10a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:57:21.891942 8436 raft_consensus.cc:2258] T 43403cf4524e4bc1876e8e0afd7ac4c3 P 021153c09e1748cda533bbdadf53d10a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:57:21.892050 8437 ts_tablet_manager.cc:1905] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:57:21.892792 8435 ts_tablet_manager.cc:1905] T 43403cf4524e4bc1876e8e0afd7ac4c3 P fd7728cbbc3c44329af3426868d6aa91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:57:21.892910 8436 ts_tablet_manager.cc:1905] T 43403cf4524e4bc1876e8e0afd7ac4c3 P 021153c09e1748cda533bbdadf53d10a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:57:21.895073 8437 ts_tablet_manager.cc:1918] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 14:57:21.895169 8437 log.cc:1196] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-0/wal/wals/43403cf4524e4bc1876e8e0afd7ac4c3
I20260927 14:57:21.895402 8437 ts_tablet_manager.cc:1939] T 43403cf4524e4bc1876e8e0afd7ac4c3 P f867c265ecbc44ec8494a1c145a40761: Deleting consensus metadata
I20260927 14:57:21.895663 8435 ts_tablet_manager.cc:1918] T 43403cf4524e4bc1876e8e0afd7ac4c3 P fd7728cbbc3c44329af3426868d6aa91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 14:57:21.895744 8435 log.cc:1196] T 43403cf4524e4bc1876e8e0afd7ac4c3 P fd7728cbbc3c44329af3426868d6aa91: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-2/wal/wals/43403cf4524e4bc1876e8e0afd7ac4c3
I20260927 14:57:21.895982 8435 ts_tablet_manager.cc:1939] T 43403cf4524e4bc1876e8e0afd7ac4c3 P fd7728cbbc3c44329af3426868d6aa91: Deleting consensus metadata
I20260927 14:57:21.896096 8436 ts_tablet_manager.cc:1918] T 43403cf4524e4bc1876e8e0afd7ac4c3 P 021153c09e1748cda533bbdadf53d10a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 14:57:21.896165 8436 log.cc:1196] T 43403cf4524e4bc1876e8e0afd7ac4c3 P 021153c09e1748cda533bbdadf53d10a: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-1/wal/wals/43403cf4524e4bc1876e8e0afd7ac4c3
I20260927 14:57:21.896373 8436 ts_tablet_manager.cc:1939] T 43403cf4524e4bc1876e8e0afd7ac4c3 P 021153c09e1748cda533bbdadf53d10a: Deleting consensus metadata
I20260927 14:57:21.896330 7888 catalog_manager.cc:4841] TS f867c265ecbc44ec8494a1c145a40761 (127.0.98.1:45889): tablet 43403cf4524e4bc1876e8e0afd7ac4c3 (table test-table [id=191598badae344f2b3ff9903675744b7]) successfully deleted
I20260927 14:57:21.896656 7887 catalog_manager.cc:4841] TS fd7728cbbc3c44329af3426868d6aa91 (127.0.98.3:40285): tablet 43403cf4524e4bc1876e8e0afd7ac4c3 (table test-table [id=191598badae344f2b3ff9903675744b7]) successfully deleted
I20260927 14:57:21.897228 7888 catalog_manager.cc:4841] TS 021153c09e1748cda533bbdadf53d10a (127.0.98.2:46375): tablet 43403cf4524e4bc1876e8e0afd7ac4c3 (table test-table [id=191598badae344f2b3ff9903675744b7]) successfully deleted
Sep 27 14:57:21 dist-test-slave-pz0h krb5kdc[7856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 14:57:21 dist-test-slave-pz0h krb5kdc[7856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:21 dist-test-slave-pz0h krb5kdc[7856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 14:57:21.926213 7950 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:21.919645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52660 (local address 127.0.98.62:40335)
0927 14:57:21.919778 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:21.919783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:21.920023 (+ 240us) server_negotiation.cc:407] Connection header received
0927 14:57:21.920158 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:21.920162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:21.920234 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:21.920307 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:21.921136 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.921670 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.922629 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.923032 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.923129 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.926108 (+ 2979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52660: BlockingRecv error: recv got EOF from 127.0.0.1:52660 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 14:57:21 dist-test-slave-pz0h krb5kdc[7856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 14:57:21.939447 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:21.931063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52666 (local address 127.0.98.62:40335)
0927 14:57:21.931190 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:21.931195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:21.931338 (+ 143us) server_negotiation.cc:407] Connection header received
0927 14:57:21.931477 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:21.931480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:21.931526 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:21.931592 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:21.932241 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.932785 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.933576 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:21.933791 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:21.933879 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:21.936403 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:21.936406 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:21.936408 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:21.936445 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:21.937740 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:21.938218 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:21.938221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:21.938223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:21.938304 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:21.938553 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:21.938555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:21.938557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:21.938742 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:21.938884 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:21.939172 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 14:57:21.939282 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 14:57:21.942237 7899 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:57:21.942701 7899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:57:21.947781 8178 tablet_service.cc:1467] Processing CreateTablet for tablet ebbf39ea15454b39bada419f179d20cb (DEFAULT_TABLE table=test-table [id=4cbd2a55a77948dea9bdd3b9079a5787]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:57:21.947783 8038 tablet_service.cc:1467] Processing CreateTablet for tablet ebbf39ea15454b39bada419f179d20cb (DEFAULT_TABLE table=test-table [id=4cbd2a55a77948dea9bdd3b9079a5787]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:57:21.947827 8318 tablet_service.cc:1467] Processing CreateTablet for tablet ebbf39ea15454b39bada419f179d20cb (DEFAULT_TABLE table=test-table [id=4cbd2a55a77948dea9bdd3b9079a5787]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:57:21.948089 8038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebbf39ea15454b39bada419f179d20cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:21.948092 8178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebbf39ea15454b39bada419f179d20cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:21.948101 8318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebbf39ea15454b39bada419f179d20cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:21.950567 8414 tablet_bootstrap.cc:492] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a: Bootstrap starting.
I20260927 14:57:21.950830 8412 tablet_bootstrap.cc:492] T ebbf39ea15454b39bada419f179d20cb P f867c265ecbc44ec8494a1c145a40761: Bootstrap starting.
I20260927 14:57:21.950925 8413 tablet_bootstrap.cc:492] T ebbf39ea15454b39bada419f179d20cb P fd7728cbbc3c44329af3426868d6aa91: Bootstrap starting.
I20260927 14:57:21.951763 8412 tablet_bootstrap.cc:654] T ebbf39ea15454b39bada419f179d20cb P f867c265ecbc44ec8494a1c145a40761: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:21.951763 8413 tablet_bootstrap.cc:654] T ebbf39ea15454b39bada419f179d20cb P fd7728cbbc3c44329af3426868d6aa91: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:21.952978 8412 tablet_bootstrap.cc:492] T ebbf39ea15454b39bada419f179d20cb P f867c265ecbc44ec8494a1c145a40761: No bootstrap required, opened a new log
I20260927 14:57:21.953074 8412 ts_tablet_manager.cc:1397] T ebbf39ea15454b39bada419f179d20cb P f867c265ecbc44ec8494a1c145a40761: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 14:57:21.953203 8414 tablet_bootstrap.cc:654] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:21.953533 8412 raft_consensus.cc:348] T ebbf39ea15454b39bada419f179d20cb P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:21.953647 8412 raft_consensus.cc:374] T ebbf39ea15454b39bada419f179d20cb P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:21.953676 8412 raft_consensus.cc:729] T ebbf39ea15454b39bada419f179d20cb P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f867c265ecbc44ec8494a1c145a40761, State: Initialized, Role: FOLLOWER
I20260927 14:57:21.953794 8412 consensus_queue.cc:260] T ebbf39ea15454b39bada419f179d20cb P f867c265ecbc44ec8494a1c145a40761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:21.954039 8412 ts_tablet_manager.cc:1428] T ebbf39ea15454b39bada419f179d20cb P f867c265ecbc44ec8494a1c145a40761: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 14:57:21.954218 8413 tablet_bootstrap.cc:492] T ebbf39ea15454b39bada419f179d20cb P fd7728cbbc3c44329af3426868d6aa91: No bootstrap required, opened a new log
I20260927 14:57:21.954312 8413 ts_tablet_manager.cc:1397] T ebbf39ea15454b39bada419f179d20cb P fd7728cbbc3c44329af3426868d6aa91: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 14:57:21.954725 8413 raft_consensus.cc:348] T ebbf39ea15454b39bada419f179d20cb P fd7728cbbc3c44329af3426868d6aa91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:21.954885 8413 raft_consensus.cc:374] T ebbf39ea15454b39bada419f179d20cb P fd7728cbbc3c44329af3426868d6aa91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:21.954923 8413 raft_consensus.cc:729] T ebbf39ea15454b39bada419f179d20cb P fd7728cbbc3c44329af3426868d6aa91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd7728cbbc3c44329af3426868d6aa91, State: Initialized, Role: FOLLOWER
I20260927 14:57:21.955049 8413 consensus_queue.cc:260] T ebbf39ea15454b39bada419f179d20cb P fd7728cbbc3c44329af3426868d6aa91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:21.955300 8413 ts_tablet_manager.cc:1428] T ebbf39ea15454b39bada419f179d20cb P fd7728cbbc3c44329af3426868d6aa91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:57:21.956441 8414 tablet_bootstrap.cc:492] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a: No bootstrap required, opened a new log
I20260927 14:57:21.956534 8414 ts_tablet_manager.cc:1397] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 14:57:21.956956 8414 raft_consensus.cc:348] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:21.957067 8414 raft_consensus.cc:374] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:21.957113 8414 raft_consensus.cc:729] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021153c09e1748cda533bbdadf53d10a, State: Initialized, Role: FOLLOWER
I20260927 14:57:21.957247 8414 consensus_queue.cc:260] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:21.957482 8414 ts_tablet_manager.cc:1428] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:57:22.074892 8432 raft_consensus.cc:482] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:57:22.075042 8432 raft_consensus.cc:504] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:22.075482 8432 leader_election.cc:288] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f867c265ecbc44ec8494a1c145a40761 (127.0.98.1:45889), fd7728cbbc3c44329af3426868d6aa91 (127.0.98.3:40285)
I20260927 14:57:22.076009 8058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebbf39ea15454b39bada419f179d20cb" candidate_uuid: "021153c09e1748cda533bbdadf53d10a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f867c265ecbc44ec8494a1c145a40761" is_pre_election: true
I20260927 14:57:22.076156 8058 raft_consensus.cc:2454] T ebbf39ea15454b39bada419f179d20cb P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021153c09e1748cda533bbdadf53d10a in term 0.
I20260927 14:57:22.076138 8338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebbf39ea15454b39bada419f179d20cb" candidate_uuid: "021153c09e1748cda533bbdadf53d10a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd7728cbbc3c44329af3426868d6aa91" is_pre_election: true
I20260927 14:57:22.076287 8338 raft_consensus.cc:2454] T ebbf39ea15454b39bada419f179d20cb P fd7728cbbc3c44329af3426868d6aa91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 021153c09e1748cda533bbdadf53d10a in term 0.
I20260927 14:57:22.076630 8131 leader_election.cc:302] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021153c09e1748cda533bbdadf53d10a, fd7728cbbc3c44329af3426868d6aa91; no voters:
I20260927 14:57:22.076802 8432 raft_consensus.cc:2781] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:57:22.076861 8432 raft_consensus.cc:482] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:57:22.076908 8432 raft_consensus.cc:3037] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:22.077881 8432 raft_consensus.cc:504] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:22.078222 8432 leader_election.cc:288] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a [CANDIDATE]: Term 1 election: Requested vote from peers f867c265ecbc44ec8494a1c145a40761 (127.0.98.1:45889), fd7728cbbc3c44329af3426868d6aa91 (127.0.98.3:40285)
I20260927 14:57:22.078585 8338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebbf39ea15454b39bada419f179d20cb" candidate_uuid: "021153c09e1748cda533bbdadf53d10a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd7728cbbc3c44329af3426868d6aa91"
I20260927 14:57:22.078590 8058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebbf39ea15454b39bada419f179d20cb" candidate_uuid: "021153c09e1748cda533bbdadf53d10a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f867c265ecbc44ec8494a1c145a40761"
I20260927 14:57:22.078701 8338 raft_consensus.cc:3037] T ebbf39ea15454b39bada419f179d20cb P fd7728cbbc3c44329af3426868d6aa91 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:22.078711 8058 raft_consensus.cc:3037] T ebbf39ea15454b39bada419f179d20cb P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:22.079582 8338 raft_consensus.cc:2454] T ebbf39ea15454b39bada419f179d20cb P fd7728cbbc3c44329af3426868d6aa91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021153c09e1748cda533bbdadf53d10a in term 1.
I20260927 14:57:22.079628 8058 raft_consensus.cc:2454] T ebbf39ea15454b39bada419f179d20cb P f867c265ecbc44ec8494a1c145a40761 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 021153c09e1748cda533bbdadf53d10a in term 1.
I20260927 14:57:22.079843 8131 leader_election.cc:302] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021153c09e1748cda533bbdadf53d10a, fd7728cbbc3c44329af3426868d6aa91; no voters:
I20260927 14:57:22.080065 8432 raft_consensus.cc:2781] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:57:22.080240 8432 raft_consensus.cc:686] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a [term 1 LEADER]: Becoming Leader. State: Replica: 021153c09e1748cda533bbdadf53d10a, State: Running, Role: LEADER
I20260927 14:57:22.080374 8432 consensus_queue.cc:237] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:22.082062 7899 catalog_manager.cc:5495] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a reported cstate change: term changed from 0 to 1, leader changed from <none> to 021153c09e1748cda533bbdadf53d10a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "021153c09e1748cda533bbdadf53d10a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } health_report { overall_health: UNKNOWN } } }
I20260927 14:57:22.119129 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:22.115412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54986 (local address 127.0.98.2:46375)
0927 14:57:22.115549 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:22.115552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:22.115568 (+ 16us) server_negotiation.cc:407] Connection header received
0927 14:57:22.115639 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:22.115641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:22.115687 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:22.115750 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:57:22.115953 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:22.116471 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:22.117463 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:22.117799 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:22.117953 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:22.118366 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:57:22.118474 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:57:22.118599 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:22.118911 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 14:57:22.119000 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:57:22.121261 8338 raft_consensus.cc:1261] T ebbf39ea15454b39bada419f179d20cb P fd7728cbbc3c44329af3426868d6aa91 [term 1 FOLLOWER]: Refusing update from remote peer 021153c09e1748cda533bbdadf53d10a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:57:22.121277 8058 raft_consensus.cc:1261] T ebbf39ea15454b39bada419f179d20cb P f867c265ecbc44ec8494a1c145a40761 [term 1 FOLLOWER]: Refusing update from remote peer 021153c09e1748cda533bbdadf53d10a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:57:22.121701 8488 consensus_queue.cc:1035] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:57:22.121886 8432 consensus_queue.cc:1035] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:57:22.129248 7897 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:57:22.129423 7897 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:57:22.131727 7897 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70a2a9a816574f60ba13f6996f2a64a7: Sending DeleteTablet for 3 replicas of tablet ebbf39ea15454b39bada419f179d20cb
I20260927 14:57:22.132566 8318 tablet_service.cc:1514] Processing DeleteTablet for tablet ebbf39ea15454b39bada419f179d20cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:57:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38396
I20260927 14:57:22.132627 8178 tablet_service.cc:1514] Processing DeleteTablet for tablet ebbf39ea15454b39bada419f179d20cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:57:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54980
I20260927 14:57:22.132784 8436 tablet_replica.cc:330] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a: stopping tablet replica
I20260927 14:57:22.132791 8038 tablet_service.cc:1514] Processing DeleteTablet for tablet ebbf39ea15454b39bada419f179d20cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:57:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51048
I20260927 14:57:22.132894 8436 raft_consensus.cc:2229] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:57:22.132961 8177 tablet_service.cc:1514] Processing DeleteTablet for tablet ebbf39ea15454b39bada419f179d20cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:57:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54980
I20260927 14:57:22.133080 8436 raft_consensus.cc:2258] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:57:22.133095 8435 tablet_replica.cc:330] T ebbf39ea15454b39bada419f179d20cb P fd7728cbbc3c44329af3426868d6aa91: stopping tablet replica
I20260927 14:57:22.133194 8435 raft_consensus.cc:2229] T ebbf39ea15454b39bada419f179d20cb P fd7728cbbc3c44329af3426868d6aa91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:57:22.133173 7897 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 14:57:22.133311 8435 raft_consensus.cc:2258] T ebbf39ea15454b39bada419f179d20cb P fd7728cbbc3c44329af3426868d6aa91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:57:22.133476 8437 tablet_replica.cc:330] T ebbf39ea15454b39bada419f179d20cb P f867c265ecbc44ec8494a1c145a40761: stopping tablet replica
I20260927 14:57:22.133599 8437 raft_consensus.cc:2229] T ebbf39ea15454b39bada419f179d20cb P f867c265ecbc44ec8494a1c145a40761 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 14:57:22.133648 7897 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:57:22.133746 8437 raft_consensus.cc:2258] T ebbf39ea15454b39bada419f179d20cb P f867c265ecbc44ec8494a1c145a40761 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:57:22.133877 8436 ts_tablet_manager.cc:1905] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:57:22.134491 8435 ts_tablet_manager.cc:1905] T ebbf39ea15454b39bada419f179d20cb P fd7728cbbc3c44329af3426868d6aa91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:57:22.134491 8437 ts_tablet_manager.cc:1905] T ebbf39ea15454b39bada419f179d20cb P f867c265ecbc44ec8494a1c145a40761: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:57:22.136592 8436 ts_tablet_manager.cc:1918] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:57:22.136674 8436 log.cc:1196] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-1/wal/wals/ebbf39ea15454b39bada419f179d20cb
I20260927 14:57:22.136916 8436 ts_tablet_manager.cc:1939] T ebbf39ea15454b39bada419f179d20cb P 021153c09e1748cda533bbdadf53d10a: Deleting consensus metadata
I20260927 14:57:22.136907 8435 ts_tablet_manager.cc:1918] T ebbf39ea15454b39bada419f179d20cb P fd7728cbbc3c44329af3426868d6aa91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:57:22.136942 8437 ts_tablet_manager.cc:1918] T ebbf39ea15454b39bada419f179d20cb P f867c265ecbc44ec8494a1c145a40761: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:57:22.136981 8435 log.cc:1196] T ebbf39ea15454b39bada419f179d20cb P fd7728cbbc3c44329af3426868d6aa91: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-2/wal/wals/ebbf39ea15454b39bada419f179d20cb
I20260927 14:57:22.137002 8437 log.cc:1196] T ebbf39ea15454b39bada419f179d20cb P f867c265ecbc44ec8494a1c145a40761: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-0/wal/wals/ebbf39ea15454b39bada419f179d20cb
I20260927 14:57:22.137210 8435 ts_tablet_manager.cc:1939] T ebbf39ea15454b39bada419f179d20cb P fd7728cbbc3c44329af3426868d6aa91: Deleting consensus metadata
I20260927 14:57:22.137218 8437 ts_tablet_manager.cc:1939] T ebbf39ea15454b39bada419f179d20cb P f867c265ecbc44ec8494a1c145a40761: Deleting consensus metadata
I20260927 14:57:22.138041 7887 catalog_manager.cc:4841] TS fd7728cbbc3c44329af3426868d6aa91 (127.0.98.3:40285): tablet ebbf39ea15454b39bada419f179d20cb (table test-table [id=4cbd2a55a77948dea9bdd3b9079a5787]) successfully deleted
I20260927 14:57:22.138150 7888 catalog_manager.cc:4841] TS 021153c09e1748cda533bbdadf53d10a (127.0.98.2:46375): tablet ebbf39ea15454b39bada419f179d20cb (table test-table [id=4cbd2a55a77948dea9bdd3b9079a5787]) successfully deleted
W20260927 14:57:22.138422 7888 catalog_manager.cc:4808] TS 021153c09e1748cda533bbdadf53d10a (127.0.98.2:46375): delete failed for tablet ebbf39ea15454b39bada419f179d20cb because the tablet was not found. No further retry: Not found: Tablet not found: ebbf39ea15454b39bada419f179d20cb
I20260927 14:57:22.139089 8038 tablet_service.cc:1467] Processing CreateTablet for tablet feeec565613043c09de445493100206a (DEFAULT_TABLE table=test-table [id=cd41b3c57acd46e8863ceb82a2671e5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:57:22.139075 8318 tablet_service.cc:1467] Processing CreateTablet for tablet feeec565613043c09de445493100206a (DEFAULT_TABLE table=test-table [id=cd41b3c57acd46e8863ceb82a2671e5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:57:22.139089 8177 tablet_service.cc:1467] Processing CreateTablet for tablet feeec565613043c09de445493100206a (DEFAULT_TABLE table=test-table [id=cd41b3c57acd46e8863ceb82a2671e5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:57:22.139359 8318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feeec565613043c09de445493100206a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:22.139360 8177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feeec565613043c09de445493100206a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:22.139360 8038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feeec565613043c09de445493100206a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:22.141464 8413 tablet_bootstrap.cc:492] T feeec565613043c09de445493100206a P fd7728cbbc3c44329af3426868d6aa91: Bootstrap starting.
I20260927 14:57:22.141906 8412 tablet_bootstrap.cc:492] T feeec565613043c09de445493100206a P f867c265ecbc44ec8494a1c145a40761: Bootstrap starting.
I20260927 14:57:22.142380 8413 tablet_bootstrap.cc:654] T feeec565613043c09de445493100206a P fd7728cbbc3c44329af3426868d6aa91: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:22.142673 8412 tablet_bootstrap.cc:654] T feeec565613043c09de445493100206a P f867c265ecbc44ec8494a1c145a40761: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:22.143208 8414 tablet_bootstrap.cc:492] T feeec565613043c09de445493100206a P 021153c09e1748cda533bbdadf53d10a: Bootstrap starting.
I20260927 14:57:22.143581 8413 tablet_bootstrap.cc:492] T feeec565613043c09de445493100206a P fd7728cbbc3c44329af3426868d6aa91: No bootstrap required, opened a new log
I20260927 14:57:22.143656 8413 ts_tablet_manager.cc:1397] T feeec565613043c09de445493100206a P fd7728cbbc3c44329af3426868d6aa91: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 14:57:22.143822 8412 tablet_bootstrap.cc:492] T feeec565613043c09de445493100206a P f867c265ecbc44ec8494a1c145a40761: No bootstrap required, opened a new log
I20260927 14:57:22.143896 8412 ts_tablet_manager.cc:1397] T feeec565613043c09de445493100206a P f867c265ecbc44ec8494a1c145a40761: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 14:57:22.144004 8414 tablet_bootstrap.cc:654] T feeec565613043c09de445493100206a P 021153c09e1748cda533bbdadf53d10a: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:22.143975 8413 raft_consensus.cc:348] T feeec565613043c09de445493100206a P fd7728cbbc3c44329af3426868d6aa91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:22.144075 8413 raft_consensus.cc:374] T feeec565613043c09de445493100206a P fd7728cbbc3c44329af3426868d6aa91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:22.144097 8413 raft_consensus.cc:729] T feeec565613043c09de445493100206a P fd7728cbbc3c44329af3426868d6aa91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd7728cbbc3c44329af3426868d6aa91, State: Initialized, Role: FOLLOWER
I20260927 14:57:22.144205 8413 consensus_queue.cc:260] T feeec565613043c09de445493100206a P fd7728cbbc3c44329af3426868d6aa91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:22.144300 8412 raft_consensus.cc:348] T feeec565613043c09de445493100206a P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:22.144408 8412 raft_consensus.cc:374] T feeec565613043c09de445493100206a P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:22.144438 8412 raft_consensus.cc:729] T feeec565613043c09de445493100206a P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f867c265ecbc44ec8494a1c145a40761, State: Initialized, Role: FOLLOWER
I20260927 14:57:22.144490 8413 ts_tablet_manager.cc:1428] T feeec565613043c09de445493100206a P fd7728cbbc3c44329af3426868d6aa91: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 14:57:22.144551 8412 consensus_queue.cc:260] T feeec565613043c09de445493100206a P f867c265ecbc44ec8494a1c145a40761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:22.145323 7888 catalog_manager.cc:4841] TS f867c265ecbc44ec8494a1c145a40761 (127.0.98.1:45889): tablet ebbf39ea15454b39bada419f179d20cb (table test-table [id=4cbd2a55a77948dea9bdd3b9079a5787]) successfully deleted
I20260927 14:57:22.145973 8412 ts_tablet_manager.cc:1428] T feeec565613043c09de445493100206a P f867c265ecbc44ec8494a1c145a40761: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 14:57:22.146916 8414 tablet_bootstrap.cc:492] T feeec565613043c09de445493100206a P 021153c09e1748cda533bbdadf53d10a: No bootstrap required, opened a new log
I20260927 14:57:22.147028 8414 ts_tablet_manager.cc:1397] T feeec565613043c09de445493100206a P 021153c09e1748cda533bbdadf53d10a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 14:57:22.147481 8414 raft_consensus.cc:348] T feeec565613043c09de445493100206a P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:22.147611 8414 raft_consensus.cc:374] T feeec565613043c09de445493100206a P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:22.147660 8414 raft_consensus.cc:729] T feeec565613043c09de445493100206a P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021153c09e1748cda533bbdadf53d10a, State: Initialized, Role: FOLLOWER
I20260927 14:57:22.147794 8414 consensus_queue.cc:260] T feeec565613043c09de445493100206a P 021153c09e1748cda533bbdadf53d10a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:22.148063 8414 ts_tablet_manager.cc:1428] T feeec565613043c09de445493100206a P 021153c09e1748cda533bbdadf53d10a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:57:22.203251 8418 raft_consensus.cc:482] T feeec565613043c09de445493100206a P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:57:22.203405 8418 raft_consensus.cc:504] T feeec565613043c09de445493100206a P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:22.203833 8418 leader_election.cc:288] T feeec565613043c09de445493100206a P f867c265ecbc44ec8494a1c145a40761 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 021153c09e1748cda533bbdadf53d10a (127.0.98.2:46375), fd7728cbbc3c44329af3426868d6aa91 (127.0.98.3:40285)
I20260927 14:57:22.204275 8198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feeec565613043c09de445493100206a" candidate_uuid: "f867c265ecbc44ec8494a1c145a40761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021153c09e1748cda533bbdadf53d10a" is_pre_election: true
I20260927 14:57:22.204313 8338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feeec565613043c09de445493100206a" candidate_uuid: "f867c265ecbc44ec8494a1c145a40761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd7728cbbc3c44329af3426868d6aa91" is_pre_election: true
I20260927 14:57:22.204433 8338 raft_consensus.cc:2454] T feeec565613043c09de445493100206a P fd7728cbbc3c44329af3426868d6aa91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f867c265ecbc44ec8494a1c145a40761 in term 0.
I20260927 14:57:22.204437 8198 raft_consensus.cc:2454] T feeec565613043c09de445493100206a P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f867c265ecbc44ec8494a1c145a40761 in term 0.
I20260927 14:57:22.204712 7992 leader_election.cc:302] T feeec565613043c09de445493100206a P f867c265ecbc44ec8494a1c145a40761 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021153c09e1748cda533bbdadf53d10a, f867c265ecbc44ec8494a1c145a40761; no voters:
I20260927 14:57:22.204921 8418 raft_consensus.cc:2781] T feeec565613043c09de445493100206a P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:57:22.204983 8418 raft_consensus.cc:482] T feeec565613043c09de445493100206a P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:57:22.205022 8418 raft_consensus.cc:3037] T feeec565613043c09de445493100206a P f867c265ecbc44ec8494a1c145a40761 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:22.205993 8418 raft_consensus.cc:504] T feeec565613043c09de445493100206a P f867c265ecbc44ec8494a1c145a40761 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:22.206328 8418 leader_election.cc:288] T feeec565613043c09de445493100206a P f867c265ecbc44ec8494a1c145a40761 [CANDIDATE]: Term 1 election: Requested vote from peers 021153c09e1748cda533bbdadf53d10a (127.0.98.2:46375), fd7728cbbc3c44329af3426868d6aa91 (127.0.98.3:40285)
I20260927 14:57:22.206702 8338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feeec565613043c09de445493100206a" candidate_uuid: "f867c265ecbc44ec8494a1c145a40761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd7728cbbc3c44329af3426868d6aa91"
I20260927 14:57:22.206702 8198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feeec565613043c09de445493100206a" candidate_uuid: "f867c265ecbc44ec8494a1c145a40761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "021153c09e1748cda533bbdadf53d10a"
I20260927 14:57:22.206856 8338 raft_consensus.cc:3037] T feeec565613043c09de445493100206a P fd7728cbbc3c44329af3426868d6aa91 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:22.206856 8198 raft_consensus.cc:3037] T feeec565613043c09de445493100206a P 021153c09e1748cda533bbdadf53d10a [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:22.208002 8198 raft_consensus.cc:2454] T feeec565613043c09de445493100206a P 021153c09e1748cda533bbdadf53d10a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f867c265ecbc44ec8494a1c145a40761 in term 1.
I20260927 14:57:22.208002 8338 raft_consensus.cc:2454] T feeec565613043c09de445493100206a P fd7728cbbc3c44329af3426868d6aa91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f867c265ecbc44ec8494a1c145a40761 in term 1.
I20260927 14:57:22.208334 7992 leader_election.cc:302] T feeec565613043c09de445493100206a P f867c265ecbc44ec8494a1c145a40761 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 021153c09e1748cda533bbdadf53d10a, f867c265ecbc44ec8494a1c145a40761; no voters:
I20260927 14:57:22.208568 8418 raft_consensus.cc:2781] T feeec565613043c09de445493100206a P f867c265ecbc44ec8494a1c145a40761 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:57:22.208663 8418 raft_consensus.cc:686] T feeec565613043c09de445493100206a P f867c265ecbc44ec8494a1c145a40761 [term 1 LEADER]: Becoming Leader. State: Replica: f867c265ecbc44ec8494a1c145a40761, State: Running, Role: LEADER
I20260927 14:57:22.208830 8418 consensus_queue.cc:237] T feeec565613043c09de445493100206a P f867c265ecbc44ec8494a1c145a40761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } }
I20260927 14:57:22.210493 7901 catalog_manager.cc:5495] T feeec565613043c09de445493100206a P f867c265ecbc44ec8494a1c145a40761 reported cstate change: term changed from 0 to 1, leader changed from <none> to f867c265ecbc44ec8494a1c145a40761 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f867c265ecbc44ec8494a1c145a40761" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f867c265ecbc44ec8494a1c145a40761" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 } health_report { overall_health: UNKNOWN } } }
I20260927 14:57:22.236950 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:22.234367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51062 (local address 127.0.98.1:45889)
0927 14:57:22.234487 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:22.234492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:22.234526 (+ 34us) server_negotiation.cc:407] Connection header received
0927 14:57:22.234574 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:22.234577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:22.234631 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:22.234686 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:57:22.234905 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:22.235417 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:22.236161 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:22.236370 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:22.236440 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:22.236615 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:57:22.236686 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:57:22.236735 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:22.236793 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 14:57:22.236835 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 14:57:22.243052 8338 raft_consensus.cc:1261] T feeec565613043c09de445493100206a P fd7728cbbc3c44329af3426868d6aa91 [term 1 FOLLOWER]: Refusing update from remote peer f867c265ecbc44ec8494a1c145a40761: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:57:22.243140 8198 raft_consensus.cc:1261] T feeec565613043c09de445493100206a P 021153c09e1748cda533bbdadf53d10a [term 1 FOLLOWER]: Refusing update from remote peer f867c265ecbc44ec8494a1c145a40761: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:57:22.243417 8418 consensus_queue.cc:1035] T feeec565613043c09de445493100206a P f867c265ecbc44ec8494a1c145a40761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd7728cbbc3c44329af3426868d6aa91" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:57:22.243585 8468 consensus_queue.cc:1035] T feeec565613043c09de445493100206a P f867c265ecbc44ec8494a1c145a40761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "021153c09e1748cda533bbdadf53d10a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:57:22.275583 7901 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:57:22.275767 7901 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:57:22.277909 7901 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70a2a9a816574f60ba13f6996f2a64a7: Sending DeleteTablet for 3 replicas of tablet feeec565613043c09de445493100206a
I20260927 14:57:22.278558 8318 tablet_service.cc:1514] Processing DeleteTablet for tablet feeec565613043c09de445493100206a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:57:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38396
I20260927 14:57:22.278573 8177 tablet_service.cc:1514] Processing DeleteTablet for tablet feeec565613043c09de445493100206a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:57:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54980
I20260927 14:57:22.278597 8038 tablet_service.cc:1514] Processing DeleteTablet for tablet feeec565613043c09de445493100206a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:57:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51048
I20260927 14:57:22.278741 8435 tablet_replica.cc:330] T feeec565613043c09de445493100206a P fd7728cbbc3c44329af3426868d6aa91: stopping tablet replica
I20260927 14:57:22.278774 8436 tablet_replica.cc:330] T feeec565613043c09de445493100206a P 021153c09e1748cda533bbdadf53d10a: stopping tablet replica
I20260927 14:57:22.278882 8435 raft_consensus.cc:2229] T feeec565613043c09de445493100206a P fd7728cbbc3c44329af3426868d6aa91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:57:22.278903 8437 tablet_replica.cc:330] T feeec565613043c09de445493100206a P f867c265ecbc44ec8494a1c145a40761: stopping tablet replica
I20260927 14:57:22.278913 8436 raft_consensus.cc:2229] T feeec565613043c09de445493100206a P 021153c09e1748cda533bbdadf53d10a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:57:22.279003 8437 raft_consensus.cc:2229] T feeec565613043c09de445493100206a P f867c265ecbc44ec8494a1c145a40761 [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:57:22.279002 8435 raft_consensus.cc:2258] T feeec565613043c09de445493100206a P fd7728cbbc3c44329af3426868d6aa91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:57:22.279009 8436 raft_consensus.cc:2258] T feeec565613043c09de445493100206a P 021153c09e1748cda533bbdadf53d10a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:57:22.279211 8437 raft_consensus.cc:2258] T feeec565613043c09de445493100206a P f867c265ecbc44ec8494a1c145a40761 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:57:22.279793 8435 ts_tablet_manager.cc:1905] T feeec565613043c09de445493100206a P fd7728cbbc3c44329af3426868d6aa91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:57:22.279908 8436 ts_tablet_manager.cc:1905] T feeec565613043c09de445493100206a P 021153c09e1748cda533bbdadf53d10a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:57:22.281919 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 7973
I20260927 14:57:22.282783 8435 ts_tablet_manager.cc:1918] T feeec565613043c09de445493100206a P fd7728cbbc3c44329af3426868d6aa91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 14:57:22.282891 8435 log.cc:1196] T feeec565613043c09de445493100206a P fd7728cbbc3c44329af3426868d6aa91: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-2/wal/wals/feeec565613043c09de445493100206a
I20260927 14:57:22.283138 8435 ts_tablet_manager.cc:1939] T feeec565613043c09de445493100206a P fd7728cbbc3c44329af3426868d6aa91: Deleting consensus metadata
I20260927 14:57:22.283283 8436 ts_tablet_manager.cc:1918] T feeec565613043c09de445493100206a P 021153c09e1748cda533bbdadf53d10a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 14:57:22.283360 8436 log.cc:1196] T feeec565613043c09de445493100206a P 021153c09e1748cda533bbdadf53d10a: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790521004237705-392-0/minicluster-data/ts-1/wal/wals/feeec565613043c09de445493100206a
I20260927 14:57:22.283592 8436 ts_tablet_manager.cc:1939] T feeec565613043c09de445493100206a P 021153c09e1748cda533bbdadf53d10a: Deleting consensus metadata
I20260927 14:57:22.283901 7887 catalog_manager.cc:4841] TS fd7728cbbc3c44329af3426868d6aa91 (127.0.98.3:40285): tablet feeec565613043c09de445493100206a (table test-table [id=cd41b3c57acd46e8863ceb82a2671e5d]) successfully deleted
I20260927 14:57:22.284516 7888 catalog_manager.cc:4841] TS 021153c09e1748cda533bbdadf53d10a (127.0.98.2:46375): tablet feeec565613043c09de445493100206a (table test-table [id=cd41b3c57acd46e8863ceb82a2671e5d]) successfully deleted
W20260927 14:57:22.289880 7888 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45889 (error 108)
I20260927 14:57:22.289968 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 8113
I20260927 14:57:22.291780 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:22.291242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45889 (local address 127.0.0.1:51074)
0927 14:57:22.291590 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:22.291668 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":60,"threads_started":1}
W20260927 14:57:22.291909 7888 catalog_manager.cc:4584] TS f867c265ecbc44ec8494a1c145a40761 (127.0.98.1:45889): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet feeec565613043c09de445493100206a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45889: connect: Connection refused (error 111)
I20260927 14:57:22.297847 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 8253
I20260927 14:57:22.304044 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 7869
2026-09-27T14:57:22Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3048 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:57:22 dist-test-slave-pz0h krb5kdc[8505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:57:22 dist-test-slave-pz0h krb5kdc[8505](info): set up 2 sockets
Sep 27 14:57:22 dist-test-slave-pz0h krb5kdc[8505](info): commencing operation
krb5kdc: starting...
W20260927 14:57:23.255312 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 14:57:23 dist-test-slave-pz0h krb5kdc[8505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:57:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:57:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:57:23.379932 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38609
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:23.496598 8518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:23.496893 8518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:23.496934 8518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:23.501075 8518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:57:23.501147 8518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:23.501165 8518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:23.501181 8518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:57:23.501195 8518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:57:23.506830 8518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:23.507956 8518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:23.508937 8518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:23.517410 8526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:23.517412 8524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:23.517390 8523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:23.517740 8518 server_base.cc:1034] running on GCE node
I20260927 14:57:23.518356 8518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:23.519596 8518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:23.520815 8518 hybrid_clock.cc:648] HybridClock initialized: now 1790521043520752 us; error 102 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T14:57:23Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1225 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:57:23 dist-test-slave-pz0h krb5kdc[8531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:57:23 dist-test-slave-pz0h krb5kdc[8531](info): set up 2 sockets
Sep 27 14:57:23 dist-test-slave-pz0h krb5kdc[8531](info): commencing operation
krb5kdc: starting...
W20260927 14:57:24.483318 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 14:57:24 dist-test-slave-pz0h krb5kdc[8531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:57:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:57:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:57:24.602746 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42869
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:24.718576 8544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:24.718896 8544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:24.718940 8544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:24.723105 8544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:57:24.723189 8544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:24.723208 8544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:24.723224 8544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:57:24.723238 8544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:57:24.728889 8544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:24.729984 8544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:24.731001 8544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 14:57:24.739686 8544 server_base.cc:1034] running on GCE node
W20260927 14:57:24.739611 8552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:24.739625 8550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:24.739632 8549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:24.740440 8544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:24.741616 8544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:24.742820 8544 hybrid_clock.cc:648] HybridClock initialized: now 1790521044742772 us; error 42 us; skew 500 ppm
Sep 27 14:57:24 dist-test-slave-pz0h krb5kdc[8531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521044, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:24.749485 8544 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:57:24.750533 8544 webserver.cc:460] Webserver started at http://127.0.98.62:45085/ using document root <none> and password file <none>
I20260927 14:57:24.751204 8544 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:24.751278 8544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:24.751482 8544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:24.753705 8544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "74928f8b154a4cf1870808340c2444d5"
format_stamp: "Formatted at 2026-09-27 14:57:24 on dist-test-slave-pz0h"
server_key: "4fb141ebed5e4dbe91b60a9b85e8c6ba"
server_key_iv: "327e86321f2a7816f7a666dc8c3ba536"
server_key_version: "encryptionkey@0"
I20260927 14:57:24.754271 8544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "74928f8b154a4cf1870808340c2444d5"
format_stamp: "Formatted at 2026-09-27 14:57:24 on dist-test-slave-pz0h"
server_key: "4fb141ebed5e4dbe91b60a9b85e8c6ba"
server_key_iv: "327e86321f2a7816f7a666dc8c3ba536"
server_key_version: "encryptionkey@0"
I20260927 14:57:24.758288 8544 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 14:57:24.760965 8559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:24.762351 8544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:57:24.762499 8544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "74928f8b154a4cf1870808340c2444d5"
format_stamp: "Formatted at 2026-09-27 14:57:24 on dist-test-slave-pz0h"
server_key: "4fb141ebed5e4dbe91b60a9b85e8c6ba"
server_key_iv: "327e86321f2a7816f7a666dc8c3ba536"
server_key_version: "encryptionkey@0"
I20260927 14:57:24.762624 8544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:24.782915 8544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:24.786065 8544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:24.786250 8544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:24.793118 8611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42869 every 8 connection(s)
I20260927 14:57:24.793119 8544 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42869
I20260927 14:57:24.794215 8544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:57:24.797060 8612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:24.799587 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 8544
I20260927 14:57:24.799719 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 14:57:24.800002 392 external_mini_cluster.cc:1428] Setting key 659b6bc1c7746794bb9c20b1afc2ec90
I20260927 14:57:24.803354 8612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5: Bootstrap starting.
I20260927 14:57:24.805963 8612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:24.806814 8612 log.cc:824] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5: Log is configured to *not* fsync() on all Append() calls
Sep 27 14:57:24 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:57:24.808957 8612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5: No bootstrap required, opened a new log
I20260927 14:57:24.813190 8612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74928f8b154a4cf1870808340c2444d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42869 } }
I20260927 14:57:24.813452 8612 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:24.813488 8612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74928f8b154a4cf1870808340c2444d5, State: Initialized, Role: FOLLOWER
I20260927 14:57:24.813956 8612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74928f8b154a4cf1870808340c2444d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42869 } }
I20260927 14:57:24.814095 8612 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:57:24.814159 8612 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:57:24.814180 8615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:24.801404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37014 (local address 127.0.98.62:42869)
0927 14:57:24.801842 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:24.801858 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:24.801893 (+ 35us) server_negotiation.cc:407] Connection header received
0927 14:57:24.802560 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:24.802578 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:24.802974 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:24.803272 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:24.804110 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:24.805103 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:24.806243 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:24.806764 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:24.806948 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:24.809455 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:24.809461 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:24.809476 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:24.809509 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:24.811202 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:24.811737 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:24.811742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:24.811747 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:24.811871 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:24.812157 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:24.812159 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:24.812160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:24.812579 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:24.812737 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:24.813022 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 14:57:24.813312 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":122,"threads_started":1}
I20260927 14:57:24.814253 8612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:24.815268 8612 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74928f8b154a4cf1870808340c2444d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42869 } }
I20260927 14:57:24.815603 8612 leader_election.cc:302] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74928f8b154a4cf1870808340c2444d5; no voters:
I20260927 14:57:24.815891 8612 leader_election.cc:288] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:57:24.815987 8617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:57:24.816143 8617 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [term 1 LEADER]: Becoming Leader. State: Replica: 74928f8b154a4cf1870808340c2444d5, State: Running, Role: LEADER
I20260927 14:57:24.816442 8617 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74928f8b154a4cf1870808340c2444d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42869 } }
I20260927 14:57:24.817205 8612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:57:24.817662 8617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74928f8b154a4cf1870808340c2444d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74928f8b154a4cf1870808340c2444d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42869 } } }
I20260927 14:57:24.817777 8617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:24.817986 8617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74928f8b154a4cf1870808340c2444d5. Latest consensus state: current_term: 1 leader_uuid: "74928f8b154a4cf1870808340c2444d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74928f8b154a4cf1870808340c2444d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42869 } } }
I20260927 14:57:24.818064 8617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:24.818442 8621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:57:24.821357 8621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:57:24.827137 8621 catalog_manager.cc:1347] Generated new cluster ID: 5edc5488b5294434a331943f04c2a0f2
I20260927 14:57:24.827224 8621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:57:24.835511 8621 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:57:24.836849 8621 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:57:24.848171 8621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5: Generated new TSK 0
I20260927 14:57:24.849072 8621 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:57:24.881686 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42869
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:25.003185 8638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:25.003492 8638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:25.003556 8638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:25.007830 8638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:25.007901 8638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:25.007988 8638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:57:25.014222 8638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:25.015759 8638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:25.016893 8638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:25.026964 8643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:25.026964 8646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:25.026997 8644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:25.027309 8638 server_base.cc:1034] running on GCE node
I20260927 14:57:25.027756 8638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:25.028432 8638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:25.029642 8638 hybrid_clock.cc:648] HybridClock initialized: now 1790521045029610 us; error 47 us; skew 500 ppm
Sep 27 14:57:25 dist-test-slave-pz0h krb5kdc[8531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:25.033044 8638 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:57:25.034041 8638 webserver.cc:460] Webserver started at http://127.0.98.1:33203/ using document root <none> and password file <none>
I20260927 14:57:25.034598 8638 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:25.034660 8638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:25.034881 8638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:25.036931 8638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/data/instance:
uuid: "200048fef772493996a49d1dbbb7d1a5"
format_stamp: "Formatted at 2026-09-27 14:57:25 on dist-test-slave-pz0h"
server_key: "b52da3ac13030f38280ab049a79e1162"
server_key_iv: "80ac204da27d2fefd79301362c927c96"
server_key_version: "encryptionkey@0"
I20260927 14:57:25.037464 8638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "200048fef772493996a49d1dbbb7d1a5"
format_stamp: "Formatted at 2026-09-27 14:57:25 on dist-test-slave-pz0h"
server_key: "b52da3ac13030f38280ab049a79e1162"
server_key_iv: "80ac204da27d2fefd79301362c927c96"
server_key_version: "encryptionkey@0"
I20260927 14:57:25.041376 8638 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 14:57:25.043998 8653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:25.045300 8638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:57:25.045425 8638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "200048fef772493996a49d1dbbb7d1a5"
format_stamp: "Formatted at 2026-09-27 14:57:25 on dist-test-slave-pz0h"
server_key: "b52da3ac13030f38280ab049a79e1162"
server_key_iv: "80ac204da27d2fefd79301362c927c96"
server_key_version: "encryptionkey@0"
I20260927 14:57:25.045553 8638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:25.071614 8638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:25.074735 8638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:25.074980 8638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:25.075598 8638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:25.076452 8638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:25.076498 8638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:25.076537 8638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:25.076560 8638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:25.085002 8638 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37935
I20260927 14:57:25.085028 8766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37935 every 8 connection(s)
I20260927 14:57:25.085943 8638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:57:25.089248 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 8638
I20260927 14:57:25.089360 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:57:25.089601 392 external_mini_cluster.cc:1428] Setting key 9f0789863929251202209a638db43b48
Sep 27 14:57:25 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 14:57:25.101851 8615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:25.087839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38395 (local address 127.0.98.62:42869)
0927 14:57:25.088002 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:25.088007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:25.088932 (+ 925us) server_negotiation.cc:407] Connection header received
0927 14:57:25.090416 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:25.090420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:25.090480 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:25.090587 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:25.091983 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:25.092940 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:25.094507 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:25.094885 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:25.094997 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:25.097852 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:25.097856 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:25.097860 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:25.097893 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:25.099621 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:25.100177 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:25.100180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:25.100182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:25.100267 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:25.100623 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:25.100626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:25.100627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:25.100834 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:25.100925 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:25.101550 (+ 625us) server_negotiation.cc:299] Negotiation successful
0927 14:57:25.101688 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 14:57:25.102766 8769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:25.088254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.1:38395)
0927 14:57:25.088766 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:25.088814 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:25.090140 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:25.090774 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:25.090783 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:25.091291 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:25.091822 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:25.091836 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:25.093137 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:25.093140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:25.094358 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:25.094370 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:25.095085 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:25.095088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:25.095265 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:25.095890 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:25.095912 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:25.097637 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:25.099758 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:25.099764 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:25.099780 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:25.100070 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:25.100352 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:25.100354 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:25.100355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:25.100509 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:25.100959 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:25.100965 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:25.101302 (+ 337us) client_negotiation.cc:769] Sending connection context
0927 14:57:25.101529 (+ 227us) client_negotiation.cc:240] Negotiation successful
0927 14:57:25.101894 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":131,"threads_started":1}
I20260927 14:57:25.104136 8767 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42869
I20260927 14:57:25.104406 8767 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:25.104974 8767 heartbeater.cc:511] Master 127.0.98.62:42869 requested a full tablet report, sending...
I20260927 14:57:25.106743 8576 ts_manager.cc:194] Registered new tserver with Master: 200048fef772493996a49d1dbbb7d1a5 (127.0.98.1:37935)
I20260927 14:57:25.108407 8576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38395
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:57:25.117918 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42869
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:25.228695 8772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:25.228992 8772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:25.229035 8772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:25.233197 8772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:25.233278 8772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:25.233340 8772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:57:25.238777 8772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:25.239856 8772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:25.240821 8772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:25.250389 8780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:25.250406 8777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:25.250391 8778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:25.250821 8772 server_base.cc:1034] running on GCE node
I20260927 14:57:25.251252 8772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:25.251859 8772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:25.253023 8772 hybrid_clock.cc:648] HybridClock initialized: now 1790521045252989 us; error 50 us; skew 500 ppm
Sep 27 14:57:25 dist-test-slave-pz0h krb5kdc[8531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:25.255756 8772 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:57:25.256620 8772 webserver.cc:460] Webserver started at http://127.0.98.2:40203/ using document root <none> and password file <none>
I20260927 14:57:25.257138 8772 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:25.257195 8772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:25.257356 8772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:25.259446 8772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/data/instance:
uuid: "ef92c17203384f6cbd0300522dfee5bd"
format_stamp: "Formatted at 2026-09-27 14:57:25 on dist-test-slave-pz0h"
server_key: "3ea23ca2256e6cb7b6874c0ae7181c7d"
server_key_iv: "d9fddcb40855fa1d649e3e1f952188a2"
server_key_version: "encryptionkey@0"
I20260927 14:57:25.259996 8772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ef92c17203384f6cbd0300522dfee5bd"
format_stamp: "Formatted at 2026-09-27 14:57:25 on dist-test-slave-pz0h"
server_key: "3ea23ca2256e6cb7b6874c0ae7181c7d"
server_key_iv: "d9fddcb40855fa1d649e3e1f952188a2"
server_key_version: "encryptionkey@0"
I20260927 14:57:25.263906 8772 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 14:57:25.266360 8787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:25.267685 8772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:57:25.267799 8772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "ef92c17203384f6cbd0300522dfee5bd"
format_stamp: "Formatted at 2026-09-27 14:57:25 on dist-test-slave-pz0h"
server_key: "3ea23ca2256e6cb7b6874c0ae7181c7d"
server_key_iv: "d9fddcb40855fa1d649e3e1f952188a2"
server_key_version: "encryptionkey@0"
I20260927 14:57:25.267912 8772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:25.280493 8772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:25.283756 8772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:25.283937 8772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:25.284529 8772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:25.285393 8772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:25.285444 8772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:25.285496 8772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:25.285526 8772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:25.294009 8772 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32881
I20260927 14:57:25.294066 8900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32881 every 8 connection(s)
I20260927 14:57:25.295025 8772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 14:57:25 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:57:25.304458 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 8772
I20260927 14:57:25.304570 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:57:25.304862 392 external_mini_cluster.cc:1428] Setting key 148816880f44469d9cad6620cd323657
I20260927 14:57:25.308354 8615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:25.296821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37657 (local address 127.0.98.62:42869)
0927 14:57:25.296949 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:25.296953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:25.297829 (+ 876us) server_negotiation.cc:407] Connection header received
0927 14:57:25.299219 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:25.299222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:25.299271 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:25.299345 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:25.300506 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:25.301024 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:25.302105 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:25.302320 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:25.302392 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:25.304845 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:25.304850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:25.304852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:25.304884 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:25.306278 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:25.306806 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:25.306811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:25.306812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:25.306900 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:25.307203 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:25.307205 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:25.307207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:25.307419 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:25.307509 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:25.308052 (+ 543us) server_negotiation.cc:299] Negotiation successful
0927 14:57:25.308194 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 14:57:25.309213 8903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:25.297214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.2:37657)
0927 14:57:25.297669 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:25.297715 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:25.299000 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:25.299512 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:25.299520 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:25.299891 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:25.300383 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:25.300397 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:25.301118 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:25.301121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:25.302003 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:25.302012 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:25.302435 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:25.302438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:25.302589 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:25.303152 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:25.303170 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:25.304637 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:25.306394 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:25.306399 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:25.306413 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:25.306688 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:25.306977 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:25.306979 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:25.306980 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:25.307123 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:25.307499 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:25.307504 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:25.307795 (+ 291us) client_negotiation.cc:769] Sending connection context
0927 14:57:25.307997 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 14:57:25.308346 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":125,"threads_started":1}
I20260927 14:57:25.310360 8901 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42869
I20260927 14:57:25.310595 8901 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:25.311116 8901 heartbeater.cc:511] Master 127.0.98.62:42869 requested a full tablet report, sending...
I20260927 14:57:25.312145 8576 ts_manager.cc:194] Registered new tserver with Master: ef92c17203384f6cbd0300522dfee5bd (127.0.98.2:32881)
I20260927 14:57:25.312719 8576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37657
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:57:25.330188 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42869
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:25.441820 8906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:25.442137 8906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:25.442198 8906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:25.446362 8906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:25.446441 8906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:25.446530 8906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:57:25.452217 8906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:25.453330 8906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:25.454413 8906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:25.464033 8912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:25.464035 8914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:25.464035 8911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:25.464390 8906 server_base.cc:1034] running on GCE node
I20260927 14:57:25.464805 8906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:25.465453 8906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:25.466640 8906 hybrid_clock.cc:648] HybridClock initialized: now 1790521045466605 us; error 46 us; skew 500 ppm
Sep 27 14:57:25 dist-test-slave-pz0h krb5kdc[8531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:25.469334 8906 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:57:25.470214 8906 webserver.cc:460] Webserver started at http://127.0.98.3:38581/ using document root <none> and password file <none>
I20260927 14:57:25.470754 8906 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:25.470856 8906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:25.471093 8906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:25.473337 8906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/data/instance:
uuid: "2eb89f669dd64c2fa2516d5704098718"
format_stamp: "Formatted at 2026-09-27 14:57:25 on dist-test-slave-pz0h"
server_key: "1147a28230e1b09a5f2f997f3b426813"
server_key_iv: "4005570a8da8ff2633403ab88d1665c5"
server_key_version: "encryptionkey@0"
I20260927 14:57:25.473907 8906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2eb89f669dd64c2fa2516d5704098718"
format_stamp: "Formatted at 2026-09-27 14:57:25 on dist-test-slave-pz0h"
server_key: "1147a28230e1b09a5f2f997f3b426813"
server_key_iv: "4005570a8da8ff2633403ab88d1665c5"
server_key_version: "encryptionkey@0"
I20260927 14:57:25.478015 8906 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260927 14:57:25.480908 8921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:25.482270 8906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:57:25.482391 8906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "2eb89f669dd64c2fa2516d5704098718"
format_stamp: "Formatted at 2026-09-27 14:57:25 on dist-test-slave-pz0h"
server_key: "1147a28230e1b09a5f2f997f3b426813"
server_key_iv: "4005570a8da8ff2633403ab88d1665c5"
server_key_version: "encryptionkey@0"
I20260927 14:57:25.482524 8906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:25.497206 8906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:25.500808 8906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:25.500991 8906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:25.501595 8906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:25.502537 8906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:25.502588 8906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:25.502643 8906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:25.502665 8906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:25.511999 8906 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42767
I20260927 14:57:25.512012 9034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42767 every 8 connection(s)
I20260927 14:57:25.513230 8906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:57:25.516754 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 8906
I20260927 14:57:25.516897 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:57:25.517235 392 external_mini_cluster.cc:1428] Setting key 3b6d88a81acb9ab07505b35511684239
Sep 27 14:57:25 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:57:25.528276 8615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:25.515324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41885 (local address 127.0.98.62:42869)
0927 14:57:25.515501 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:25.515505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:25.516435 (+ 930us) server_negotiation.cc:407] Connection header received
0927 14:57:25.518209 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:25.518213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:25.518270 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:25.518347 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:25.519831 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:25.520355 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:25.521530 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:25.521751 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:25.521841 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:25.524548 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:25.524552 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:25.524555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:25.524583 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:25.525916 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:25.526546 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:25.526550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:25.526551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:25.526649 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:25.527114 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:25.527118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:25.527119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:25.527329 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:25.527421 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:25.528014 (+ 593us) server_negotiation.cc:299] Negotiation successful
0927 14:57:25.528139 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 14:57:25.529125 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:25.515712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.3:41885)
0927 14:57:25.516253 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:25.516304 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:25.517910 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:25.518558 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:25.518567 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:25.519094 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:25.519692 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:25.519708 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:25.520485 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:25.520488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:25.521397 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:25.521408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:25.521905 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:25.521909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:25.522086 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:25.522708 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:25.522727 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:25.524337 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:25.526084 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:25.526090 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:25.526106 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:25.526413 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:25.526772 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:25.526775 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:25.526776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:25.526995 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:25.527430 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:25.527436 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:25.527723 (+ 287us) client_negotiation.cc:769] Sending connection context
0927 14:57:25.527899 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 14:57:25.528215 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":144,"threads_started":1}
I20260927 14:57:25.530407 9035 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42869
I20260927 14:57:25.530692 9035 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:25.531350 9035 heartbeater.cc:511] Master 127.0.98.62:42869 requested a full tablet report, sending...
I20260927 14:57:25.532598 8576 ts_manager.cc:194] Registered new tserver with Master: 2eb89f669dd64c2fa2516d5704098718 (127.0.98.3:42767)
I20260927 14:57:25.533381 8576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41885
I20260927 14:57:25.543601 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 14:57:25.615823 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 8638
I20260927 14:57:25.621167 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 8772
I20260927 14:57:25.626283 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 8906
I20260927 14:57:25.631121 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 8544
I20260927 14:57:25.636581 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42869
--webserver_interface=127.0.98.62
--webserver_port=45085
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:25.753156 9044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:25.753465 9044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:25.753527 9044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:25.757877 9044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:57:25.757992 9044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:25.758020 9044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:25.758044 9044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:57:25.758064 9044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:57:25.764288 9044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:25.765420 9044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:25.766470 9044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:25.776311 9052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:25.776290 9050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:25.776486 9044 server_base.cc:1034] running on GCE node
W20260927 14:57:25.776290 9049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:25.777258 9044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:25.778503 9044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:25.779670 9044 hybrid_clock.cc:648] HybridClock initialized: now 1790521045779641 us; error 49 us; skew 500 ppm
Sep 27 14:57:25 dist-test-slave-pz0h krb5kdc[8531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:25.782398 9044 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:57:25.783408 9044 webserver.cc:460] Webserver started at http://127.0.98.62:45085/ using document root <none> and password file <none>
I20260927 14:57:25.783967 9044 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:25.784051 9044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:25.788435 9044 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:57:25.790774 9059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:25.792258 9044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:57:25.792425 9044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "74928f8b154a4cf1870808340c2444d5"
format_stamp: "Formatted at 2026-09-27 14:57:24 on dist-test-slave-pz0h"
server_key: "4fb141ebed5e4dbe91b60a9b85e8c6ba"
server_key_iv: "327e86321f2a7816f7a666dc8c3ba536"
server_key_version: "encryptionkey@0"
I20260927 14:57:25.793062 9044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:25.805994 9044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:25.809103 9044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:25.809296 9044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:25.816416 9044 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42869
I20260927 14:57:25.816701 9111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42869 every 8 connection(s)
I20260927 14:57:25.817596 9044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:57:25.822019 9112 sys_catalog.cc:263] Verifying existing consensus state
I20260927 14:57:25.823336 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 9044
I20260927 14:57:25.823761 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37935
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42869
--webserver_port=33203
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:57:25.823959 9112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5: Bootstrap starting.
I20260927 14:57:25.840639 9112 log.cc:824] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5: Log is configured to *not* fsync() on all Append() calls
I20260927 14:57:25.847561 9112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 14:57:25.847919 9112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5: Bootstrap complete.
I20260927 14:57:25.852085 9112 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74928f8b154a4cf1870808340c2444d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42869 } }
I20260927 14:57:25.852512 9112 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74928f8b154a4cf1870808340c2444d5, State: Initialized, Role: FOLLOWER
I20260927 14:57:25.853041 9112 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74928f8b154a4cf1870808340c2444d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42869 } }
I20260927 14:57:25.853178 9112 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:57:25.853241 9112 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:57:25.853332 9112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [term 1 FOLLOWER]: Advancing to term 2
I20260927 14:57:25.855870 9112 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74928f8b154a4cf1870808340c2444d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42869 } }
I20260927 14:57:25.856282 9112 leader_election.cc:302] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74928f8b154a4cf1870808340c2444d5; no voters:
I20260927 14:57:25.856647 9112 leader_election.cc:288] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 14:57:25.856810 9116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 14:57:25.857103 9116 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [term 2 LEADER]: Becoming Leader. State: Replica: 74928f8b154a4cf1870808340c2444d5, State: Running, Role: LEADER
I20260927 14:57:25.857542 9116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74928f8b154a4cf1870808340c2444d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42869 } }
I20260927 14:57:25.858057 9112 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:57:25.858701 9116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "74928f8b154a4cf1870808340c2444d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74928f8b154a4cf1870808340c2444d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42869 } } }
I20260927 14:57:25.858898 9116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:25.859135 9116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74928f8b154a4cf1870808340c2444d5. Latest consensus state: current_term: 2 leader_uuid: "74928f8b154a4cf1870808340c2444d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74928f8b154a4cf1870808340c2444d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42869 } } }
I20260927 14:57:25.859229 9116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5 [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:25.859483 9124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:57:25.864246 9124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:57:25.864974 9124 catalog_manager.cc:1259] Loaded cluster ID: 5edc5488b5294434a331943f04c2a0f2
I20260927 14:57:25.865051 9124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:57:25.867408 9124 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:57:25.868253 9124 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 74928f8b154a4cf1870808340c2444d5: Loaded TSK: 0
I20260927 14:57:25.869542 9124 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 14:57:25.943717 9114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:25.943981 9114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:25.944025 9114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:25.948202 9114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:25.948283 9114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:25.948345 9114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:57:25.953994 9114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37935
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33203
--tserver_master_addrs=127.0.98.62:42869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:25.955224 9114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:25.956257 9114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:25.966100 9141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:25.966096 9138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:25.966567 9139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:25.966782 9114 server_base.cc:1034] running on GCE node
I20260927 14:57:25.967258 9114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:25.967948 9114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:25.969141 9114 hybrid_clock.cc:648] HybridClock initialized: now 1790521045969101 us; error 62 us; skew 500 ppm
Sep 27 14:57:25 dist-test-slave-pz0h krb5kdc[8531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521045, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:25.972141 9114 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 14:57:25.973335 9114 webserver.cc:460] Webserver started at http://127.0.98.1:33203/ using document root <none> and password file <none>
I20260927 14:57:25.973920 9114 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:25.974001 9114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:25.978168 9114 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:57:25.980497 9148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:25.981849 9114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 14:57:25.981978 9114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "200048fef772493996a49d1dbbb7d1a5"
format_stamp: "Formatted at 2026-09-27 14:57:25 on dist-test-slave-pz0h"
server_key: "b52da3ac13030f38280ab049a79e1162"
server_key_iv: "80ac204da27d2fefd79301362c927c96"
server_key_version: "encryptionkey@0"
I20260927 14:57:25.982607 9114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:26.000458 9114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:26.003527 9114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:26.003714 9114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:26.004311 9114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:26.005208 9114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:26.005255 9114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:26.005295 9114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:26.005311 9114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:26.014367 9114 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37935
I20260927 14:57:26.014395 9261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37935 every 8 connection(s)
I20260927 14:57:26.015420 9114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:57:26.020349 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 9114
I20260927 14:57:26.020767 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:32881
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42869
--webserver_port=40203
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 14:57:26 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:26 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:26.030778 9265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:26.017866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.1:47581)
0927 14:57:26.018392 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:26.018441 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:26.019794 (+ 1353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:26.021221 (+ 1427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:26.021233 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:26.021758 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:26.022406 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:26.022424 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.024179 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.024183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:26.025171 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:26.025185 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.026439 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.026442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:26.026629 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:26.027399 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:26.027421 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:26.029902 (+ 2481us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":124,"threads_started":1}
W20260927 14:57:26.031337 9264 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:26.017714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47581 (local address 127.0.98.62:42869)
0927 14:57:26.018296 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:26.018317 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:26.018586 (+ 269us) server_negotiation.cc:407] Connection header received
0927 14:57:26.020266 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:26.020278 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:26.020645 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:26.021018 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:26.022668 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.023971 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.025352 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.026207 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.026425 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:26.030268 (+ 3843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47581: BlockingRecv error: recv got EOF from 127.0.98.1:47581 (error 108)
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":163,"threads_started":1}
W20260927 14:57:26.031631 9262 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42869 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:42869: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 14:57:26 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:26 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:26.037595 9265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:26.032110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.1:46989)
0927 14:57:26.032298 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:26.032311 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:26.032388 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:26.032648 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:26.032650 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:26.032795 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:26.032914 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:26.032919 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.034016 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.034019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:26.034579 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:26.034589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.035208 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.035211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:26.035298 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:26.035820 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:26.035831 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:26.037482 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114}
W20260927 14:57:26.037796 9264 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:26.032228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46989 (local address 127.0.98.62:42869)
0927 14:57:26.032359 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:26.032364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:26.032380 (+ 16us) server_negotiation.cc:407] Connection header received
0927 14:57:26.032464 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:26.032467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:26.032519 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:26.032614 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:26.033018 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.033863 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.034694 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.035087 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.035190 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:26.037673 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46989: BlockingRecv error: recv got EOF from 127.0.98.1:46989 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 14:57:26 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:26 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:26.043502 9265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:26.038315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.1:44359)
0927 14:57:26.038493 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:26.038507 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:26.038577 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:26.038849 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:26.038851 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:26.038982 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:26.039062 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:26.039067 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.040081 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.040084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:26.040600 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:26.040608 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.041158 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.041161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:26.041222 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:26.041731 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:26.041743 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:26.043393 (+ 1650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108}
W20260927 14:57:26.043696 9264 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:26.038438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44359 (local address 127.0.98.62:42869)
0927 14:57:26.038553 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:26.038557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:26.038572 (+ 15us) server_negotiation.cc:407] Connection header received
0927 14:57:26.038663 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:26.038666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:26.038714 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:26.038828 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:26.039154 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.039968 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.040703 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.041058 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.041156 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:26.043566 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44359: BlockingRecv error: recv got EOF from 127.0.98.1:44359 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 14:57:26.043859 9262 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 14:57:26.137444 9266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:26.137748 9266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:26.137794 9266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:26.142021 9266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:26.142112 9266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:26.142174 9266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:57:26.147766 9266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:32881
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40203
--tserver_master_addrs=127.0.98.62:42869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:26.148937 9266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:26.149968 9266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:26.159524 9272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:26.159524 9274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:26.159528 9271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:26.159861 9266 server_base.cc:1034] running on GCE node
I20260927 14:57:26.160290 9266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:26.160916 9266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:26.162091 9266 hybrid_clock.cc:648] HybridClock initialized: now 1790521046162070 us; error 32 us; skew 500 ppm
Sep 27 14:57:26 dist-test-slave-pz0h krb5kdc[8531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521046, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:26.164803 9266 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 14:57:26.165714 9266 webserver.cc:460] Webserver started at http://127.0.98.2:40203/ using document root <none> and password file <none>
I20260927 14:57:26.166280 9266 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:26.166364 9266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:26.170439 9266 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:57:26.172688 9281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:26.174077 9266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:57:26.174216 9266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "ef92c17203384f6cbd0300522dfee5bd"
format_stamp: "Formatted at 2026-09-27 14:57:25 on dist-test-slave-pz0h"
server_key: "3ea23ca2256e6cb7b6874c0ae7181c7d"
server_key_iv: "d9fddcb40855fa1d649e3e1f952188a2"
server_key_version: "encryptionkey@0"
I20260927 14:57:26.174773 9266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:26.194229 9266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:26.197600 9266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:26.197796 9266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:26.198410 9266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:26.199301 9266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:26.199348 9266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:26.199402 9266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:26.199431 9266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:26.208105 9266 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32881
I20260927 14:57:26.208138 9394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32881 every 8 connection(s)
I20260927 14:57:26.209182 9266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 14:57:26 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:26.218876 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 9266
Sep 27 14:57:26 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:26.219288 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42767
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42869
--webserver_port=38581
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:26.220485 9264 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:26.211002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42899 (local address 127.0.98.62:42869)
0927 14:57:26.211123 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:26.211126 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:26.211963 (+ 837us) server_negotiation.cc:407] Connection header received
0927 14:57:26.213361 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:26.213365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:26.213430 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:26.213511 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:26.214858 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.215391 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.216619 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.216831 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.216919 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:26.220355 (+ 3436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42899: BlockingRecv error: recv got EOF from 127.0.98.2:42899 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 14:57:26.221143 9397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:26.211365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.2:42899)
0927 14:57:26.211808 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:26.211852 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:26.213132 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:26.213702 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:26.213709 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:26.214115 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:26.214644 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:26.214659 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.215536 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.215539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:26.216479 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:26.216490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.216991 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.216995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:26.217170 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:26.217766 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:26.217784 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:26.220189 (+ 2405us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":117,"threads_started":1}
W20260927 14:57:26.221603 9395 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42869 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:42869: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 14:57:26 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:26 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:26.227214 9397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:26.222040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.2:33891)
0927 14:57:26.222289 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:26.222303 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:26.222400 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:26.222640 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:26.222643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:26.222842 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:26.222950 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:26.222956 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.223660 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.223664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:26.224205 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:26.224212 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.224605 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.224608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:26.224700 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:26.225167 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:26.225181 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:26.227072 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181}
W20260927 14:57:26.227427 9264 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:26.222164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33891 (local address 127.0.98.62:42869)
0927 14:57:26.222279 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:26.222283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:26.222351 (+ 68us) server_negotiation.cc:407] Connection header received
0927 14:57:26.222501 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:26.222504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:26.222548 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:26.222612 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:26.223059 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.223562 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.224306 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.224510 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.224594 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:26.227208 (+ 2614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33891: BlockingRecv error: recv got EOF from 127.0.98.2:33891 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 27 14:57:26 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:26 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:26.233294 9397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:26.227813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.2:51981)
0927 14:57:26.227962 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:26.227977 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:26.228067 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:26.228303 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:26.228306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:26.228473 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:26.228584 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:26.228591 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.229296 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.229299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:26.230206 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:26.230216 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.230656 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.230659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:26.230741 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:26.231231 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:26.231245 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:26.233149 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 14:57:26.233390 9264 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:26.227917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51981 (local address 127.0.98.62:42869)
0927 14:57:26.228019 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:26.228022 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:26.228034 (+ 12us) server_negotiation.cc:407] Connection header received
0927 14:57:26.228158 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:26.228160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:26.228198 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:26.228260 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:26.228691 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.229194 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.230337 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.230557 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.230637 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:26.233283 (+ 2646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51981: BlockingRecv error: recv got EOF from 127.0.98.2:51981 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260927 14:57:26.233664 9395 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 14:57:26.332615 9398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:26.332911 9398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:26.332953 9398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:26.337116 9398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:26.337199 9398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:26.337260 9398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:57:26.342759 9398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42767
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38581
--tserver_master_addrs=127.0.98.62:42869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:26.343937 9398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:26.344956 9398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:26.354490 9404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:26.354494 9406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:26.354493 9403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:26.354552 9398 server_base.cc:1034] running on GCE node
I20260927 14:57:26.355269 9398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:26.355937 9398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:26.357103 9398 hybrid_clock.cc:648] HybridClock initialized: now 1790521046357084 us; error 31 us; skew 500 ppm
Sep 27 14:57:26 dist-test-slave-pz0h krb5kdc[8531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521046, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:26.359915 9398 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 14:57:26.360904 9398 webserver.cc:460] Webserver started at http://127.0.98.3:38581/ using document root <none> and password file <none>
I20260927 14:57:26.361438 9398 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:26.361526 9398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:26.365700 9398 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:57:26.367892 9413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:26.369179 9398 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 14:57:26.369305 9398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "2eb89f669dd64c2fa2516d5704098718"
format_stamp: "Formatted at 2026-09-27 14:57:25 on dist-test-slave-pz0h"
server_key: "1147a28230e1b09a5f2f997f3b426813"
server_key_iv: "4005570a8da8ff2633403ab88d1665c5"
server_key_version: "encryptionkey@0"
I20260927 14:57:26.369869 9398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:26.384493 9398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:26.387794 9398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:26.387981 9398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:26.388588 9398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:26.389460 9398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:26.389519 9398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:26.389578 9398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:26.389605 9398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:26.398576 9398 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42767
I20260927 14:57:26.398598 9526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42767 every 8 connection(s)
I20260927 14:57:26.399684 9398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:57:26.405977 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 9398
Sep 27 14:57:26 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:26 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 14:57:26.412221 9264 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:26.401617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50233 (local address 127.0.98.62:42869)
0927 14:57:26.401785 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:26.401789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:26.402707 (+ 918us) server_negotiation.cc:407] Connection header received
0927 14:57:26.404097 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:26.404101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:26.404159 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:26.404245 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:26.405566 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.406119 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.407753 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.407978 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.408079 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:26.412091 (+ 4012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50233: BlockingRecv error: recv got EOF from 127.0.98.3:50233 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 14:57:26.413015 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:26.402022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.3:50233)
0927 14:57:26.402533 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:26.402579 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:26.403872 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:26.404435 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:26.404443 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:26.404878 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:26.405404 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:26.405419 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.406254 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.406257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:26.407631 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:26.407644 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.408136 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.408140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:26.408341 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:26.408960 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:26.408981 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:26.411952 (+ 2971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":126,"threads_started":1}
W20260927 14:57:26.413539 9527 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42869 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:42869: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260927 14:57:26.416159 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:26.407046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37024 (local address 127.0.98.62:42869)
0927 14:57:26.407300 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:26.407304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:26.407318 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:57:26.407352 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:26.407355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:26.407402 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:26.407507 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:26.408260 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.408942 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.409844 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.410123 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.410224 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:26.411885 (+ 1661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:26.411891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:26.411908 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:26.411941 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:26.414122 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:26.414532 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:26.414538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:26.414545 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:26.414661 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:26.414945 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:26.414949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:26.414951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:26.415329 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:26.415507 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:26.415797 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 14:57:26.416026 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":92,"threads_started":1}
Sep 27 14:57:26 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:26 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:26.419236 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:26.413982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.3:60689)
0927 14:57:26.414102 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:26.414117 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:26.414217 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:26.414432 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:26.414435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:26.414578 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:26.414674 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:26.414679 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.415408 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.415411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:26.415976 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:26.415988 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.416468 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.416472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:26.416574 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:26.417031 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:26.417045 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:26.419093 (+ 2048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 14:57:26.419348 9264 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:26.414049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60689 (local address 127.0.98.62:42869)
0927 14:57:26.414155 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:26.414159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:26.414172 (+ 13us) server_negotiation.cc:407] Connection header received
0927 14:57:26.414300 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:26.414304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:26.414342 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:26.414405 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:26.414779 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.415310 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.416090 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.416356 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.416448 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:26.419245 (+ 2797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60689: BlockingRecv error: recv got EOF from 127.0.98.3:60689 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 14:57:26 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:26 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:26.425117 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:26.419843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.3:56219)
0927 14:57:26.419965 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:26.419977 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:26.420044 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:26.420231 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:26.420233 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:26.420357 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:26.420439 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:26.420445 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.421134 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.421137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:26.421664 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:26.421672 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.422330 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.422333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:26.422400 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:26.422975 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:26.422994 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:26.425004 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 14:57:26.425297 9264 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:26.419928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56219 (local address 127.0.98.62:42869)
0927 14:57:26.420054 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:26.420058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:26.420088 (+ 30us) server_negotiation.cc:407] Connection header received
0927 14:57:26.420120 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:26.420122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:26.420157 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:26.420212 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:26.420532 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.421025 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.421868 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:26.422188 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:26.422287 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:26.425166 (+ 2879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56219: BlockingRecv error: recv got EOF from 127.0.98.3:56219 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 14:57:26.425374 9527 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 14:57:27 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:27 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:27.050514 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:27.044603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.1:46837)
0927 14:57:27.044879 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:27.044893 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:27.044988 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:27.045350 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:27.045353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:27.045539 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:27.045646 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:27.045653 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:27.046463 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:27.046466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:27.047111 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:27.047122 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:27.047634 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:27.047638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:27.047719 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:27.048300 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:27.048313 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:27.050316 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":78,"threads_started":1}
W20260927 14:57:27.050601 9533 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:27.044798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46837 (local address 127.0.98.62:42869)
0927 14:57:27.045053 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:27.045057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:27.045071 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:57:27.045118 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:27.045120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:27.045163 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:27.045262 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:27.045792 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:27.046339 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:27.047229 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:27.047480 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:27.047584 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:27.050470 (+ 2886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46837: BlockingRecv error: recv got EOF from 127.0.98.1:46837 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":95,"threads_started":1}
Sep 27 14:57:27 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:27 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:27.240108 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:27.234442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.2:48035)
0927 14:57:27.234706 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:27.234722 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:27.234863 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:27.235108 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:27.235110 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:27.235290 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:27.235399 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:27.235404 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:27.236186 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:27.236189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:27.236776 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:27.236785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:27.237380 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:27.237385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:27.237476 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:27.238082 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:27.238098 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:27.239975 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":80,"threads_started":1}
W20260927 14:57:27.240229 9533 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:27.234550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48035 (local address 127.0.98.62:42869)
0927 14:57:27.234712 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:27.234716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:27.234812 (+ 96us) server_negotiation.cc:407] Connection header received
0927 14:57:27.234949 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:27.234951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:27.235003 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:27.235083 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:27.235534 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:27.236059 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:27.236919 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:27.237222 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:27.237332 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:27.240103 (+ 2771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48035: BlockingRecv error: recv got EOF from 127.0.98.2:48035 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 14:57:27 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:27 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:27.431126 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:27.426027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.3:57793)
0927 14:57:27.426284 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:27.426296 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:27.426407 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:27.426659 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:27.426661 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:27.426870 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:27.426989 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:27.426995 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:27.427712 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:27.427714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:27.428269 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:27.428277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:27.428656 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:27.428658 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:27.428738 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:27.429200 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:27.429212 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:27.430976 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":79,"threads_started":1}
W20260927 14:57:27.431237 9533 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:27.426105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57793 (local address 127.0.98.62:42869)
0927 14:57:27.426246 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:27.426250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:27.426367 (+ 117us) server_negotiation.cc:407] Connection header received
0927 14:57:27.426506 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:27.426508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:27.426562 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:27.426628 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:27.427086 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:27.427607 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:27.428357 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:27.428574 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:27.428642 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:27.431131 (+ 2489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57793: BlockingRecv error: recv got EOF from 127.0.98.3:57793 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 14:57:28 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:28 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:28.056681 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:28.051487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.1:48285)
0927 14:57:28.051762 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:28.051775 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:28.051877 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:28.052118 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:28.052120 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:28.052301 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:28.052399 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:28.052405 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:28.053193 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:28.053196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:28.053739 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:28.053746 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:28.054156 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:28.054159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:28.054231 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:28.054732 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:28.054746 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:28.056526 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":104,"threads_started":1}
W20260927 14:57:28.056777 9537 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:28.051577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48285 (local address 127.0.98.62:42869)
0927 14:57:28.051798 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:28.051802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:28.051833 (+ 31us) server_negotiation.cc:407] Connection header received
0927 14:57:28.051961 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:28.051963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:28.052014 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:28.052090 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:28.052538 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:28.053096 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:28.053829 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:28.054060 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:28.054135 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:28.056664 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48285: BlockingRecv error: recv got EOF from 127.0.98.1:48285 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":58,"threads_started":1}
Sep 27 14:57:28 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:28 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:28.246214 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:28.240943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.2:60091)
0927 14:57:28.241287 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:28.241303 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:28.241432 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:28.241704 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:28.241707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:28.241943 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:28.242060 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:28.242068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:28.242871 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:28.242874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:28.243423 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:28.243432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:28.243853 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:28.243856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:28.243937 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:28.244405 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:28.244416 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:28.246028 (+ 1612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":112,"threads_started":1}
W20260927 14:57:28.246282 9537 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:28.241097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60091 (local address 127.0.98.62:42869)
0927 14:57:28.241238 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:28.241243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:28.241387 (+ 144us) server_negotiation.cc:407] Connection header received
0927 14:57:28.241529 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:28.241532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:28.241585 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:28.241657 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:28.242185 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:28.242733 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:28.243532 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:28.243751 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:28.243837 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:28.246180 (+ 2343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60091: BlockingRecv error: recv got EOF from 127.0.98.2:60091 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 14:57:28 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:28 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:28.437114 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:28.431930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.3:33635)
0927 14:57:28.432251 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:28.432267 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:28.432388 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:28.432638 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:28.432641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:28.432839 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:28.432927 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:28.432933 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:28.433656 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:28.433660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:28.434212 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:28.434220 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:28.434643 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:28.434646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:28.434728 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:28.435229 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:28.435241 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:28.436968 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":107,"threads_started":1}
W20260927 14:57:28.437265 9537 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:28.432137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33635 (local address 127.0.98.62:42869)
0927 14:57:28.432268 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:28.432273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:28.432337 (+ 64us) server_negotiation.cc:407] Connection header received
0927 14:57:28.432483 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:28.432486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:28.432537 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:28.432600 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:28.433031 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:28.433550 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:28.434315 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:28.434539 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:28.434625 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:28.437147 (+ 2522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33635: BlockingRecv error: recv got EOF from 127.0.98.3:33635 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 14:57:29 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:29 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:29.062775 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:29.057439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.1:58699)
0927 14:57:29.057714 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:29.057728 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:29.057832 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:29.058062 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:29.058064 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:29.058305 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:29.058400 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:29.058407 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:29.059345 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:29.059348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:29.059939 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:29.059948 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:29.060380 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:29.060384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:29.060461 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:29.060928 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:29.060940 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:29.062631 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":75,"threads_started":1}
W20260927 14:57:29.062937 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:29.057534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58699 (local address 127.0.98.62:42869)
0927 14:57:29.057779 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:29.057782 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:29.057796 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:57:29.057907 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:29.057909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:29.057952 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:29.058035 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:29.058524 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:29.059196 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:29.060050 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:29.060280 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:29.060363 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:29.062770 (+ 2407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58699: BlockingRecv error: recv got EOF from 127.0.98.1:58699 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":91,"threads_started":1}
Sep 27 14:57:29 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:29 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:29.252470 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:29.247140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.2:34593)
0927 14:57:29.247456 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:29.247472 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:29.247586 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:29.247853 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:29.247856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:29.248067 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:29.248188 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:29.248195 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:29.248930 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:29.248933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:29.249486 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:29.249492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:29.249951 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:29.249954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:29.250020 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:29.250546 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:29.250562 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:29.252333 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":83,"threads_started":1}
W20260927 14:57:29.252601 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:29.247216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34593 (local address 127.0.98.62:42869)
0927 14:57:29.247403 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:29.247407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:29.247548 (+ 141us) server_negotiation.cc:407] Connection header received
0927 14:57:29.247675 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:29.247678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:29.247733 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:29.247802 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:29.248304 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:29.248838 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:29.249614 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:29.249841 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:29.249928 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:29.252480 (+ 2552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34593: BlockingRecv error: recv got EOF from 127.0.98.2:34593 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 14:57:29 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:29 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:29.443342 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:29.438053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.3:46009)
0927 14:57:29.438340 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:29.438358 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:29.438463 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:29.438699 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:29.438701 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:29.438937 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:29.439051 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:29.439058 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:29.439968 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:29.439971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:29.440551 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:29.440561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:29.440957 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:29.440959 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:29.441018 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:29.441456 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:29.441466 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:29.443208 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":89,"threads_started":1}
W20260927 14:57:29.443447 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:29.438192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46009 (local address 127.0.98.62:42869)
0927 14:57:29.438339 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:29.438342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:29.438424 (+ 82us) server_negotiation.cc:407] Connection header received
0927 14:57:29.438554 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:29.438556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:29.438607 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:29.438669 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:29.439173 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:29.439874 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:29.440662 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:29.440871 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:29.440948 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:29.443347 (+ 2399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46009: BlockingRecv error: recv got EOF from 127.0.98.3:46009 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 14:57:30 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:30 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:30.073967 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:30.063640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.1:45451)
0927 14:57:30.064071 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:30.064085 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:30.064209 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:30.064501 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:30.064503 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:30.064692 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:30.064810 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:30.064819 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:30.065709 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:30.065714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:30.066488 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:30.066505 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:30.067054 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:30.067058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:30.067146 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:30.071565 (+ 4419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:30.071587 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:30.073832 (+ 2245us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":88,"threads_started":1}
W20260927 14:57:30.074193 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:30.063768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45451 (local address 127.0.98.62:42869)
0927 14:57:30.063995 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:30.063999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:30.064171 (+ 172us) server_negotiation.cc:407] Connection header received
0927 14:57:30.064306 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:30.064310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:30.064369 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:30.064460 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:30.064924 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:30.065464 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:30.066643 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:30.066909 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:30.067009 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:30.074049 (+ 7040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45451: BlockingRecv error: recv got EOF from 127.0.98.1:45451 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":70,"threads_started":1}
Sep 27 14:57:30 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:30 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:30.258600 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:30.253250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.2:48657)
0927 14:57:30.253540 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:30.253559 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:30.253672 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:30.253912 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:30.253915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:30.254123 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:30.254217 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:30.254222 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:30.255053 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:30.255058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:30.255625 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:30.255634 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:30.256063 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:30.256067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:30.256147 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:30.256715 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:30.256729 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:30.258480 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":82,"threads_started":1}
W20260927 14:57:30.258867 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:30.253354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48657 (local address 127.0.98.62:42869)
0927 14:57:30.253496 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:30.253500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:30.253630 (+ 130us) server_negotiation.cc:407] Connection header received
0927 14:57:30.253761 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:30.253763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:30.253811 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:30.253882 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:30.254336 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:30.254952 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:30.255734 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:30.255958 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:30.256051 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:30.258648 (+ 2597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48657: BlockingRecv error: recv got EOF from 127.0.98.2:48657 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 14:57:30 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:30 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:30.449030 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:30.444148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.3:57183)
0927 14:57:30.444399 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:30.444412 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:30.444514 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:30.444738 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:30.444740 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:30.444917 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:30.445010 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:30.445015 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:30.445693 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:30.445695 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:30.446250 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:30.446257 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:30.446622 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:30.446625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:30.446686 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:30.447194 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:30.447207 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:30.448896 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":75,"threads_started":1}
W20260927 14:57:30.449144 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:30.444261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57183 (local address 127.0.98.62:42869)
0927 14:57:30.444405 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:30.444409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:30.444477 (+ 68us) server_negotiation.cc:407] Connection header received
0927 14:57:30.444591 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:30.444593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:30.444637 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:30.444714 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:30.445092 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:30.445610 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:30.446333 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:30.446545 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:30.446611 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:30.449040 (+ 2429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57183: BlockingRecv error: recv got EOF from 127.0.98.3:57183 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 14:57:31 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:31 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:31.080601 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:31.074943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.1:42869)
0927 14:57:31.075242 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:31.075257 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:31.075438 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:31.075778 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:31.075781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:31.075951 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:31.076048 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:31.076053 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:31.076841 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:31.076844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:31.077665 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:31.077675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:31.078116 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:31.078118 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:31.078177 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:31.078694 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:31.078706 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:31.080478 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":91,"threads_started":1}
W20260927 14:57:31.080765 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:31.075024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42869 (local address 127.0.98.62:42869)
0927 14:57:31.075275 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:31.075279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:31.075314 (+ 35us) server_negotiation.cc:407] Connection header received
0927 14:57:31.075576 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:31.075579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:31.075634 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:31.075719 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:31.076142 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:31.076715 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:31.077761 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:31.078013 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:31.078093 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:31.080617 (+ 2524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42869: BlockingRecv error: recv got EOF from 127.0.98.1:42869 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":83,"threads_started":1}
Sep 27 14:57:31 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:31 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:31.264978 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:31.259399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.2:34755)
0927 14:57:31.259701 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:31.259727 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:31.259846 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:31.260090 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:31.260092 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:31.260292 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:31.260398 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:31.260404 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:31.261161 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:31.261165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:31.261746 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:31.261755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:31.262180 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:31.262184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:31.262262 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:31.262769 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:31.262781 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:31.264822 (+ 2041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":90,"threads_started":1}
W20260927 14:57:31.265098 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:31.259627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34755 (local address 127.0.98.62:42869)
0927 14:57:31.259753 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:31.259758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:31.259788 (+ 30us) server_negotiation.cc:407] Connection header received
0927 14:57:31.259938 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:31.259941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:31.259989 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:31.260054 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:31.260518 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:31.261050 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:31.261860 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:31.262078 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:31.262162 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:31.264979 (+ 2817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34755: BlockingRecv error: recv got EOF from 127.0.98.2:34755 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 14:57:31 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:31 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:31.455928 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:31.449964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.3:39091)
0927 14:57:31.450204 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:31.450217 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:31.450311 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:31.450612 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:31.450614 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:31.450781 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:31.450912 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:31.450920 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:31.452003 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:31.452006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:31.452671 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:31.452684 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:31.453136 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:31.453139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:31.453223 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:31.453675 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:31.453687 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:31.455796 (+ 2109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":65,"threads_started":1}
W20260927 14:57:31.456043 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:31.450075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39091 (local address 127.0.98.62:42869)
0927 14:57:31.450220 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:31.450224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:31.450274 (+ 50us) server_negotiation.cc:407] Connection header received
0927 14:57:31.450405 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:31.450409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:31.450469 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:31.450601 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:31.451016 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:31.451862 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:31.452794 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:31.453020 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:31.453110 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:31.455943 (+ 2833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39091: BlockingRecv error: recv got EOF from 127.0.98.3:39091 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 14:57:32 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:32 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:32.087195 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:32.081361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.1:58035)
0927 14:57:32.081638 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:32.081652 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:32.081750 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:32.082236 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:32.082239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:32.082425 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:32.082559 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:32.082566 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:32.083535 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:32.083538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:32.084119 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:32.084128 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:32.084591 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:32.084594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:32.084662 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:32.085176 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:32.085187 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:32.087051 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":76,"threads_started":1}
W20260927 14:57:32.087380 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:32.081450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58035 (local address 127.0.98.62:42869)
0927 14:57:32.081867 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:32.081871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:32.081894 (+ 23us) server_negotiation.cc:407] Connection header received
0927 14:57:32.081953 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:32.081956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:32.082020 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:32.082150 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:32.082773 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:32.083397 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:32.084239 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:32.084471 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:32.084554 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:32.087236 (+ 2682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58035: BlockingRecv error: recv got EOF from 127.0.98.1:58035 (error 108)
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":73,"threads_started":1}
Sep 27 14:57:32 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:32 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:32.271795 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:32.266036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.2:60331)
0927 14:57:32.266384 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:32.266398 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:32.266529 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:32.266883 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:32.266886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:32.267083 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:32.267197 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:32.267203 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:32.267932 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:32.267935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:32.268488 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:32.268495 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:32.268981 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:32.268984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:32.269056 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:32.269726 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:32.269739 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:32.271641 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":126,"threads_started":1}
W20260927 14:57:32.271890 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:32.266165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60331 (local address 127.0.98.62:42869)
0927 14:57:32.266366 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:32.266371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:32.266514 (+ 143us) server_negotiation.cc:407] Connection header received
0927 14:57:32.266649 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:32.266652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:32.266714 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:32.266838 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:32.267312 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:32.267843 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:32.268621 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:32.268847 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:32.268954 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:32.271793 (+ 2839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60331: BlockingRecv error: recv got EOF from 127.0.98.2:60331 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 27 14:57:32 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:32 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:32.462777 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:32.456799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.3:40787)
0927 14:57:32.457095 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:32.457111 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:32.457243 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:32.457488 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:32.457491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:32.457878 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:32.458126 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:32.458137 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:32.459039 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:32.459043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:32.459798 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:32.459810 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:32.460243 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:32.460247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:32.460335 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:32.460863 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:32.460876 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:32.462611 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":90,"threads_started":1}
W20260927 14:57:32.462905 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:32.456933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40787 (local address 127.0.98.62:42869)
0927 14:57:32.457095 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:32.457100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:32.457185 (+ 85us) server_negotiation.cc:407] Connection header received
0927 14:57:32.457333 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:32.457336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:32.457383 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:32.457453 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:32.458279 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:32.458914 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:32.459909 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:32.460140 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:32.460227 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:32.462771 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40787: BlockingRecv error: recv got EOF from 127.0.98.3:40787 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 14:57:33 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:33 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:33.093544 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:33.088045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.1:57263)
0927 14:57:33.088359 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:33.088372 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:33.088468 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:33.088713 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:33.088715 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:33.088892 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:33.088989 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:33.088994 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:33.089714 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:33.089717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:33.090332 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:33.090343 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:33.091027 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:33.091031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:33.091121 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:33.091645 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:33.091663 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:33.093424 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":85,"threads_started":1}
W20260927 14:57:33.093703 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:33.088168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57263 (local address 127.0.98.62:42869)
0927 14:57:33.088409 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:33.088412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:33.088427 (+ 15us) server_negotiation.cc:407] Connection header received
0927 14:57:33.088550 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:33.088552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:33.088596 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:33.088674 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:33.089088 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:33.089616 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:33.090455 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:33.090743 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:33.090859 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:33.093566 (+ 2707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57263: BlockingRecv error: recv got EOF from 127.0.98.1:57263 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":57,"threads_started":1}
Sep 27 14:57:33 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:33 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:33.279903 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:33.272989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.2:48073)
0927 14:57:33.273545 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:33.273564 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:33.273720 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:33.274024 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:33.274027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:33.274279 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:33.274391 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:33.274398 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:33.275426 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:33.275429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:33.276023 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:33.276035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:33.276743 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:33.276746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:33.276823 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:33.277379 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:33.277393 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:33.279745 (+ 2352us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":117,"threads_started":1}
W20260927 14:57:33.280333 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:33.273134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48073 (local address 127.0.98.62:42869)
0927 14:57:33.273338 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:33.273342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:33.273705 (+ 363us) server_negotiation.cc:407] Connection header received
0927 14:57:33.273819 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:33.273822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:33.273875 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:33.273942 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:33.274542 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:33.275251 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:33.276197 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:33.276537 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:33.276655 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:33.280163 (+ 3508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48073: BlockingRecv error: recv got EOF from 127.0.98.2:48073 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 14:57:33 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:33 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:33.468820 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:33.463635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.3:54077)
0927 14:57:33.463979 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:33.463991 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:33.464114 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:33.464359 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:33.464361 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:33.464556 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:33.464650 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:33.464656 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:33.465402 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:33.465406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:33.465954 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:33.465962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:33.466373 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:33.466376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:33.466456 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:33.466996 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:33.467008 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:33.468694 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":75,"threads_started":1}
W20260927 14:57:33.468969 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:33.463744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54077 (local address 127.0.98.62:42869)
0927 14:57:33.463867 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:33.463872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:33.464064 (+ 192us) server_negotiation.cc:407] Connection header received
0927 14:57:33.464212 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:33.464214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:33.464258 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:33.464321 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:33.464755 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:33.465297 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:33.466060 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:33.466273 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:33.466356 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:33.468843 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54077: BlockingRecv error: recv got EOF from 127.0.98.3:54077 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 27 14:57:34 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:34 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:34.099856 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:34.094333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.1:59377)
0927 14:57:34.094673 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:34.094688 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:34.094824 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:34.095142 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:34.095145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:34.095335 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:34.095430 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:34.095436 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:34.096203 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:34.096207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:34.096800 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:34.096809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:34.097271 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:34.097274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:34.097361 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:34.097938 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:34.097949 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:34.099722 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":76,"threads_started":1}
W20260927 14:57:34.100083 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:34.094480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59377 (local address 127.0.98.62:42869)
0927 14:57:34.094674 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:34.094679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:34.094742 (+ 63us) server_negotiation.cc:407] Connection header received
0927 14:57:34.094975 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:34.094978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:34.095030 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:34.095115 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:34.095537 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:34.096090 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:34.096913 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:34.097163 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:34.097250 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:34.099923 (+ 2673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59377: BlockingRecv error: recv got EOF from 127.0.98.1:59377 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":53,"threads_started":1}
Sep 27 14:57:34 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:34 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:34.286521 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:34.280856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.2:45337)
0927 14:57:34.281160 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:34.281177 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:34.281301 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:34.281543 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:34.281546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:34.281749 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:34.281865 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:34.281871 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:34.282781 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:34.282785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:34.283514 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:34.283524 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:34.283998 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:34.284002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:34.284094 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:34.284563 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:34.284575 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:34.286382 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":88,"threads_started":1}
W20260927 14:57:34.286772 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:34.281022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45337 (local address 127.0.98.62:42869)
0927 14:57:34.281163 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:34.281168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:34.281250 (+ 82us) server_negotiation.cc:407] Connection header received
0927 14:57:34.281390 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:34.281393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:34.281441 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:34.281508 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:34.281978 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:34.282679 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:34.283616 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:34.283883 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:34.283972 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:34.286552 (+ 2580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45337: BlockingRecv error: recv got EOF from 127.0.98.2:45337 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 14:57:34 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:34 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:34.475286 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:34.469738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.3:58947)
0927 14:57:34.470012 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:34.470025 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:34.470126 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:34.470361 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:34.470363 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:34.470536 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:34.470633 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:34.470638 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:34.471393 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:34.471397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:34.471993 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:34.472004 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:34.472446 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:34.472449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:34.472530 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:34.473030 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:34.473043 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:34.475154 (+ 2111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":98,"threads_started":1}
W20260927 14:57:34.475406 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:34.469851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58947 (local address 127.0.98.62:42869)
0927 14:57:34.469979 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:34.469983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:34.470087 (+ 104us) server_negotiation.cc:407] Connection header received
0927 14:57:34.470205 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:34.470207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:34.470255 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:34.470335 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:34.470723 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:34.471273 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:34.472122 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:34.472340 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:34.472428 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:34.475303 (+ 2875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58947: BlockingRecv error: recv got EOF from 127.0.98.3:58947 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 14:57:35 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:35 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:35.110674 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:35.100735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.1:36265)
0927 14:57:35.101042 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:35.101059 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:35.101187 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:35.101463 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:35.101466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:35.101659 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:35.101761 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:35.101767 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:35.102573 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:35.102576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:35.103172 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:35.103181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:35.103607 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:35.103610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:35.103669 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:35.108622 (+ 4953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:35.108638 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:35.110539 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":80,"threads_started":1}
W20260927 14:57:35.110908 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:35.100839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36265 (local address 127.0.98.62:42869)
0927 14:57:35.101091 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:35.101095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:35.101130 (+ 35us) server_negotiation.cc:407] Connection header received
0927 14:57:35.101271 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:35.101274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:35.101320 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:35.101399 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:35.101885 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:35.102450 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:35.103285 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:35.103511 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:35.103597 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:35.110733 (+ 7136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36265: BlockingRecv error: recv got EOF from 127.0.98.1:36265 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":82,"threads_started":1}
Sep 27 14:57:35 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:35 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:35.292619 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:35.287243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.2:33893)
0927 14:57:35.287516 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:35.287528 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:35.287637 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:35.287882 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:35.287885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:35.288083 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:35.288174 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:35.288180 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:35.288912 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:35.288916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:35.289478 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:35.289487 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:35.289938 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:35.289941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:35.290024 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:35.290517 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:35.290528 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:35.292448 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":93,"threads_started":1}
W20260927 14:57:35.292721 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:35.287317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33893 (local address 127.0.98.62:42869)
0927 14:57:35.287448 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:35.287452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:35.287599 (+ 147us) server_negotiation.cc:407] Connection header received
0927 14:57:35.287727 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:35.287730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:35.287782 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:35.287850 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:35.288287 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:35.288810 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:35.289593 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:35.289813 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:35.289898 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:35.292616 (+ 2718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33893: BlockingRecv error: recv got EOF from 127.0.98.2:33893 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 14:57:35 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:35 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:35.481015 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:35.476000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.3:44551)
0927 14:57:35.476254 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:35.476267 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:35.476364 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:35.476572 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:35.476574 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:35.476742 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:35.476835 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:35.476840 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:35.477527 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:35.477531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:35.478108 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:35.478118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:35.478475 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:35.478478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:35.478539 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:35.479050 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:35.479063 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:35.480864 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":94,"threads_started":1}
W20260927 14:57:35.481103 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:35.476062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44551 (local address 127.0.98.62:42869)
0927 14:57:35.476197 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:35.476201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:35.476327 (+ 126us) server_negotiation.cc:407] Connection header received
0927 14:57:35.476432 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:35.476434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:35.476478 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:35.476539 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:35.476914 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:35.477422 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:35.478192 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:35.478393 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:35.478457 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:35.481004 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44551: BlockingRecv error: recv got EOF from 127.0.98.3:44551 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 14:57:36 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:36 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:36.118022 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:36.111562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.1:37487)
0927 14:57:36.111979 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:36.111994 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:36.112106 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:36.112355 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:36.112358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:36.112539 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:36.112643 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:36.112648 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:36.113483 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:36.113490 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:36.114084 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:36.114093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:36.114598 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:36.114607 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:36.114686 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:36.115560 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:36.115573 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:36.117891 (+ 2318us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":85,"threads_started":1}
W20260927 14:57:36.118220 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:36.111611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37487 (local address 127.0.98.62:42869)
0927 14:57:36.111875 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:36.111880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:36.112076 (+ 196us) server_negotiation.cc:407] Connection header received
0927 14:57:36.112182 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:36.112185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:36.112239 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:36.112320 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:36.112804 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:36.113348 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:36.114235 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:36.114481 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:36.114570 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:36.118074 (+ 3504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37487: BlockingRecv error: recv got EOF from 127.0.98.1:37487 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":68,"threads_started":1}
Sep 27 14:57:36 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:36 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:36.299084 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:36.293430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.2:41465)
0927 14:57:36.293730 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:36.293743 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:36.293841 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:36.294049 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:36.294050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:36.294249 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:36.294344 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:36.294349 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:36.295087 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:36.295089 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:36.295646 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:36.295653 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:36.296153 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:36.296156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:36.296224 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:36.296822 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:36.296835 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:36.298883 (+ 2048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":76,"threads_started":1}
W20260927 14:57:36.299152 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:36.293487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41465 (local address 127.0.98.62:42869)
0927 14:57:36.293659 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:36.293664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:36.293837 (+ 173us) server_negotiation.cc:407] Connection header received
0927 14:57:36.293905 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:36.293907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:36.293953 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:36.294020 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:36.294446 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:36.295001 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:36.295766 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:36.295992 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:36.296099 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:36.299046 (+ 2947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41465: BlockingRecv error: recv got EOF from 127.0.98.2:41465 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 14:57:36 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:36 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:36.487836 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:36.482219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.3:57605)
0927 14:57:36.482477 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:36.482490 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:36.482604 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:36.482874 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:36.482876 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:36.483088 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:36.483188 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:36.483196 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:36.484012 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:36.484015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:36.484577 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:36.484586 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:36.485201 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:36.485204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:36.485281 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:36.485792 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:36.485805 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:36.487702 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":80,"threads_started":1}
W20260927 14:57:36.488103 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:36.482332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57605 (local address 127.0.98.62:42869)
0927 14:57:36.482452 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:36.482456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:36.482567 (+ 111us) server_negotiation.cc:407] Connection header received
0927 14:57:36.482699 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:36.482702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:36.482753 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:36.482855 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:36.483323 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:36.483885 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:36.484787 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:36.485073 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:36.485165 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:36.487899 (+ 2734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57605: BlockingRecv error: recv got EOF from 127.0.98.3:57605 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 14:57:37 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:37 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:37.124521 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:37.119019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.1:42573)
0927 14:57:37.119308 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:37.119323 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:37.119418 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:37.119656 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:37.119658 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:37.119839 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:37.119933 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:37.119937 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:37.120745 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:37.120749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:37.121318 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:37.121327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:37.121795 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:37.121798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:37.121859 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:37.122445 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:37.122461 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:37.124377 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":79,"threads_started":1}
W20260927 14:57:37.124644 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:37.119129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42573 (local address 127.0.98.62:42869)
0927 14:57:37.119349 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:37.119353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:37.119367 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:57:37.119501 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:37.119504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:37.119546 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:37.119628 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:37.120067 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:37.120613 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:37.121451 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:37.121691 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:37.121779 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:37.124515 (+ 2736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42573: BlockingRecv error: recv got EOF from 127.0.98.1:42573 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":67,"threads_started":1}
Sep 27 14:57:37 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:37 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:37.305584 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:37.299903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.2:60977)
0927 14:57:37.300267 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:37.300279 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:37.300382 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:37.300712 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:37.300715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:37.300944 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:37.301044 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:37.301051 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:37.301801 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:37.301804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:37.302367 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:37.302375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:37.302864 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:37.302868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:37.302936 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:37.303479 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:37.303491 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:37.305459 (+ 1968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":81,"threads_started":1}
W20260927 14:57:37.305747 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:37.299996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60977 (local address 127.0.98.62:42869)
0927 14:57:37.300163 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:37.300167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:37.300357 (+ 190us) server_negotiation.cc:407] Connection header received
0927 14:57:37.300557 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:37.300560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:37.300609 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:37.300667 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:37.301147 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:37.301668 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:37.302493 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:37.302727 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:37.302839 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:37.305606 (+ 2767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60977: BlockingRecv error: recv got EOF from 127.0.98.2:60977 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 27 14:57:37 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:37 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:37.494150 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:37.488698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.3:42111)
0927 14:57:37.488965 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:37.488978 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:37.489081 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:37.489317 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:37.489319 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:37.489497 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:37.489591 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:37.489596 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:37.490470 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:37.490474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:37.491069 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:37.491081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:37.491528 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:37.491530 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:37.491590 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:37.492098 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:37.492109 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:37.494001 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":87,"threads_started":1}
W20260927 14:57:37.494306 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:37.488804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42111 (local address 127.0.98.62:42869)
0927 14:57:37.488964 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:37.488968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:37.489063 (+ 95us) server_negotiation.cc:407] Connection header received
0927 14:57:37.489160 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:37.489163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:37.489212 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:37.489290 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:37.489731 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:37.490331 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:37.491202 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:37.491433 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:37.491517 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:37.494184 (+ 2667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42111: BlockingRecv error: recv got EOF from 127.0.98.3:42111 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 27 14:57:38 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:38 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:38.130852 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:38.125288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.1:53797)
0927 14:57:38.125543 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:38.125557 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:38.125661 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:38.125919 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:38.125921 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:38.126106 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:38.126217 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:38.126222 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:38.127083 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:38.127086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:38.127653 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:38.127661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:38.128134 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:38.128136 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:38.128209 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:38.128771 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:38.128785 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:38.130672 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":76,"threads_started":1}
W20260927 14:57:38.131022 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:38.125361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53797 (local address 127.0.98.62:42869)
0927 14:57:38.125585 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:38.125589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:38.125603 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:57:38.125727 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:38.125729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:38.125771 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:38.125846 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:38.126345 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:38.126965 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:38.127787 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:38.128025 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:38.128114 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:38.130886 (+ 2772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53797: BlockingRecv error: recv got EOF from 127.0.98.1:53797 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":69,"threads_started":1}
Sep 27 14:57:38 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:38 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:38.312951 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:38.306492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.2:36547)
0927 14:57:38.306818 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:38.306836 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:38.306943 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:38.307257 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:38.307259 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:38.307444 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:38.307547 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:38.307553 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:38.308434 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:38.308437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:38.309014 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:38.309023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:38.309804 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:38.309810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:38.309922 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:38.310633 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:38.310656 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:38.312815 (+ 2159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":80,"threads_started":1}
W20260927 14:57:38.313151 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:38.306692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36547 (local address 127.0.98.62:42869)
0927 14:57:38.306885 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:38.306888 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:38.306903 (+ 15us) server_negotiation.cc:407] Connection header received
0927 14:57:38.307070 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:38.307073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:38.307123 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:38.307208 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:38.307733 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:38.308290 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:38.309226 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:38.309546 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:38.309692 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:38.313022 (+ 3330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36547: BlockingRecv error: recv got EOF from 127.0.98.2:36547 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 27 14:57:38 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:38 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:38.500379 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:38.495001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.3:45443)
0927 14:57:38.495272 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:38.495286 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:38.495394 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:38.495622 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:38.495624 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:38.495795 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:38.495890 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:38.495896 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:38.496648 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:38.496651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:38.497313 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:38.497323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:38.497798 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:38.497803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:38.497884 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:38.498389 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:38.498401 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:38.500218 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
W20260927 14:57:38.500476 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:38.495142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45443 (local address 127.0.98.62:42869)
0927 14:57:38.495292 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:38.495296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:38.495352 (+ 56us) server_negotiation.cc:407] Connection header received
0927 14:57:38.495469 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:38.495471 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:38.495522 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:38.495590 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:38.495983 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:38.496551 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:38.497462 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:38.497690 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:38.497781 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:38.500373 (+ 2592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45443: BlockingRecv error: recv got EOF from 127.0.98.3:45443 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 14:57:39 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:39 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:39.137203 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:39.131793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.1:47025)
0927 14:57:39.132085 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:39.132098 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:39.132202 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:39.132433 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:39.132435 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:39.132612 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:39.132714 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:39.132719 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:39.133511 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:39.133515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:39.134096 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:39.134105 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:39.134604 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:39.134608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:39.134686 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:39.135294 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:39.135306 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:39.137075 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":95,"threads_started":1}
W20260927 14:57:39.137431 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:39.131893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47025 (local address 127.0.98.62:42869)
0927 14:57:39.132141 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:39.132145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:39.132160 (+ 15us) server_negotiation.cc:407] Connection header received
0927 14:57:39.132284 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:39.132287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:39.132330 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:39.132408 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:39.132834 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:39.133391 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:39.134221 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:39.134469 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:39.134555 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:39.137288 (+ 2733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47025: BlockingRecv error: recv got EOF from 127.0.98.1:47025 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":57,"threads_started":1}
Sep 27 14:57:39 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:39 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:39.319808 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:39.313949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.2:37355)
0927 14:57:39.314223 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:39.314240 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:39.314351 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:39.314655 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:39.314658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:39.314927 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:39.315043 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:39.315050 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:39.315975 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:39.315978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:39.316550 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:39.316559 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:39.316975 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:39.316977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:39.317038 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:39.317582 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:39.317596 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:39.319627 (+ 2031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":72,"threads_started":1}
W20260927 14:57:39.319921 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:39.314262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37355 (local address 127.0.98.62:42869)
0927 14:57:39.314409 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:39.314414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:39.314430 (+ 16us) server_negotiation.cc:407] Connection header received
0927 14:57:39.314474 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:39.314477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:39.314517 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:39.314607 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:39.315176 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:39.315871 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:39.316660 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:39.316886 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:39.316961 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:39.319818 (+ 2857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37355: BlockingRecv error: recv got EOF from 127.0.98.2:37355 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 14:57:39 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:39 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:39.507021 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:39.501328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.3:58293)
0927 14:57:39.501682 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:39.501697 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:39.501830 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:39.502094 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:39.502097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:39.502312 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:39.502406 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:39.502412 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:39.503222 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:39.503225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:39.503789 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:39.503798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:39.504234 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:39.504237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:39.504333 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:39.504861 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:39.504878 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:39.506873 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":113,"threads_started":1}
W20260927 14:57:39.507115 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:39.501485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58293 (local address 127.0.98.62:42869)
0927 14:57:39.501673 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:39.501677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:39.501785 (+ 108us) server_negotiation.cc:407] Connection header received
0927 14:57:39.501929 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:39.501932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:39.501981 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:39.502051 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:39.502528 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:39.503115 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:39.503908 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:39.504126 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:39.504211 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:39.507009 (+ 2798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58293: BlockingRecv error: recv got EOF from 127.0.98.3:58293 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 14:57:40 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:40 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:40.149583 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:40.137989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.1:33199)
0927 14:57:40.138244 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:40.138257 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:40.138344 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:40.138566 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:40.138568 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:40.138750 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:40.138880 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:40.138886 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:40.139637 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:40.139640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:40.140212 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:40.140222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:40.140611 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:40.140614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:40.140686 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:40.147563 (+ 6877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:40.147581 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:40.149448 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":75,"threads_started":1}
W20260927 14:57:40.149722 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:40.138065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33199 (local address 127.0.98.62:42869)
0927 14:57:40.138368 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:40.138371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:40.138385 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:57:40.138420 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:40.138422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:40.138462 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:40.138545 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:40.138974 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:40.139502 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:40.140313 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:40.140529 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:40.140601 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:40.149598 (+ 8997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33199: BlockingRecv error: recv got EOF from 127.0.98.1:33199 (error 108)
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":74,"threads_started":1}
Sep 27 14:57:40 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:40 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:40.326112 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:40.320780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.2:58845)
0927 14:57:40.321089 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:40.321107 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:40.321233 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:40.321484 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:40.321487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:40.321688 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:40.321783 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:40.321789 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:40.322522 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:40.322525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:40.323153 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:40.323162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:40.323616 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:40.323620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:40.323708 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:40.324197 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:40.324209 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:40.325932 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":93,"threads_started":1}
W20260927 14:57:40.326211 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:40.320924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58845 (local address 127.0.98.62:42869)
0927 14:57:40.321097 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:40.321101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:40.321192 (+ 91us) server_negotiation.cc:407] Connection header received
0927 14:57:40.321329 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:40.321332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:40.321379 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:40.321447 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:40.321896 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:40.322421 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:40.323274 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:40.323490 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:40.323576 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:40.326096 (+ 2520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58845: BlockingRecv error: recv got EOF from 127.0.98.2:58845 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 27 14:57:40 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:40 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:40.513697 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:40.508054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.3:57461)
0927 14:57:40.508347 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:40.508362 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:40.508483 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:40.508732 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:40.508735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:40.508929 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:40.509022 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:40.509028 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:40.509775 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:40.509779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:40.510329 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:40.510337 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:40.510766 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:40.510770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:40.510897 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:40.511376 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:40.511389 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:40.513434 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":91,"threads_started":1}
W20260927 14:57:40.513787 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:40.508189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57461 (local address 127.0.98.62:42869)
0927 14:57:40.508347 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:40.508351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:40.508436 (+ 85us) server_negotiation.cc:407] Connection header received
0927 14:57:40.508574 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:40.508577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:40.508625 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:40.508690 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:40.509149 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:40.509668 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:40.510442 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:40.510660 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:40.510745 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:40.513596 (+ 2851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57461: BlockingRecv error: recv got EOF from 127.0.98.3:57461 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 14:57:41 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:41 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:41.156183 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:41.150614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.1:50209)
0927 14:57:41.150906 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:41.150922 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:41.151030 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:41.151264 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:41.151266 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:41.151442 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:41.151539 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:41.151545 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:41.152310 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:41.152313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:41.152899 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:41.152908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:41.153319 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:41.153321 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:41.153381 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:41.153881 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:41.153897 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:41.156048 (+ 2151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":79,"threads_started":1}
W20260927 14:57:41.156332 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:41.150725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50209 (local address 127.0.98.62:42869)
0927 14:57:41.150999 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:41.151003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:41.151020 (+ 17us) server_negotiation.cc:407] Connection header received
0927 14:57:41.151115 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:41.151118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:41.151166 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:41.151250 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:41.151628 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:41.152210 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:41.153004 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:41.153226 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:41.153312 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:41.156210 (+ 2898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50209: BlockingRecv error: recv got EOF from 127.0.98.1:50209 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":93,"threads_started":1}
Sep 27 14:57:41 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 14:57:41 dist-test-slave-pz0h krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 14:57:41.332213 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:41.327083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42869 (local address 127.0.98.2:42053)
0927 14:57:41.327366 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:41.327382 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:41.327500 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:41.327749 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:41.327752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:41.327948 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:41.328039 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:41.328044 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:41.328771 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:41.328774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:41.329330 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:41.329338 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:41.329754 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:41.329758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:41.329833 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:41.330335 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:41.330346 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:41.332046 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42869: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":79,"threads_started":1}
W20260927 14:57:41.332309 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:41.327207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42053 (local address 127.0.98.62:42869)
0927 14:57:41.327371 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:41.327375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:41.327460 (+ 85us) server_negotiation.cc:407] Connection header received
0927 14:57:41.327593 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:41.327596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:41.327643 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:41.327713 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:41.328158 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:41.328671 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:41.329439 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:41.329653 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:41.329736 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:41.332203 (+ 2467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42053: BlockingRecv error: recv got EOF from 127.0.98.2:42053 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 14:57:41.414232 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 9114
I20260927 14:57:41.419013 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 9266
I20260927 14:57:41.423877 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 9398
I20260927 14:57:41.428203 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 9044
2026-09-27T14:57:41Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17897 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T14:57:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:57:41Z Disabled control of system clock
I20260927 14:57:41.456655 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46111
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 14:57:41.566483 9594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:41.566783 9594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:41.566839 9594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:41.570973 9594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:57:41.571054 9594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:41.571069 9594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:41.571084 9594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:57:41.571098 9594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:57:41.576988 9594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46111
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:41.578109 9594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:41.579546 9594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:41.588784 9599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:41.588896 9600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:41.588963 9602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:41.589273 9594 server_base.cc:1034] running on GCE node
I20260927 14:57:41.589967 9594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:41.591197 9594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:41.592396 9594 hybrid_clock.cc:648] HybridClock initialized: now 1790521061592368 us; error 49 us; skew 500 ppm
I20260927 14:57:41.594417 9594 webserver.cc:460] Webserver started at http://127.0.98.62:34031/ using document root <none> and password file <none>
I20260927 14:57:41.595078 9594 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:41.595147 9594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:41.595348 9594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:41.597464 9594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "1719781fad924b3d9315c164f04c9db8"
format_stamp: "Formatted at 2026-09-27 14:57:41 on dist-test-slave-pz0h"
server_key: "a2d6d8ff913a6084ec9495157e4071b6"
server_key_iv: "0e25cd9e670cb653b89867bc30777067"
server_key_version: "encryptionkey@0"
I20260927 14:57:41.598040 9594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "1719781fad924b3d9315c164f04c9db8"
format_stamp: "Formatted at 2026-09-27 14:57:41 on dist-test-slave-pz0h"
server_key: "a2d6d8ff913a6084ec9495157e4071b6"
server_key_iv: "0e25cd9e670cb653b89867bc30777067"
server_key_version: "encryptionkey@0"
I20260927 14:57:41.602025 9594 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 14:57:41.604712 9608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:41.606057 9594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:57:41.606196 9594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "1719781fad924b3d9315c164f04c9db8"
format_stamp: "Formatted at 2026-09-27 14:57:41 on dist-test-slave-pz0h"
server_key: "a2d6d8ff913a6084ec9495157e4071b6"
server_key_iv: "0e25cd9e670cb653b89867bc30777067"
server_key_version: "encryptionkey@0"
I20260927 14:57:41.606317 9594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:41.624325 9594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:41.625070 9594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:41.625252 9594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:41.632285 9594 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46111
I20260927 14:57:41.632288 9660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46111 every 8 connection(s)
I20260927 14:57:41.633303 9594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:57:41.636548 9661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:41.642378 9661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1719781fad924b3d9315c164f04c9db8: Bootstrap starting.
I20260927 14:57:41.642572 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 9594
I20260927 14:57:41.642683 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 14:57:41.642958 392 external_mini_cluster.cc:1428] Setting key 88fcf2d5bb104aaec6bebf3f546a5b9c
I20260927 14:57:41.644984 9661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1719781fad924b3d9315c164f04c9db8: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:41.645704 9661 log.cc:824] T 00000000000000000000000000000000 P 1719781fad924b3d9315c164f04c9db8: Log is configured to *not* fsync() on all Append() calls
I20260927 14:57:41.647750 9661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1719781fad924b3d9315c164f04c9db8: No bootstrap required, opened a new log
I20260927 14:57:41.651710 9661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1719781fad924b3d9315c164f04c9db8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1719781fad924b3d9315c164f04c9db8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46111 } }
I20260927 14:57:41.651923 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:41.644248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33272 (local address 127.0.98.62:46111)
0927 14:57:41.644880 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:41.644898 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:41.644933 (+ 35us) server_negotiation.cc:407] Connection header received
0927 14:57:41.645596 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:41.645614 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:41.646046 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:41.646422 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:41.646770 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:41.647758 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:41.648633 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:41.649150 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:41.649295 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:41.650150 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:41.650155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:41.650168 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:57:41.650561 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:41.650620 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:57:41.650640 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:41.650724 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:41.650890 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 14:57:41.651155 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":114,"threads_started":1}
I20260927 14:57:41.651969 9661 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1719781fad924b3d9315c164f04c9db8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:41.652019 9661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1719781fad924b3d9315c164f04c9db8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1719781fad924b3d9315c164f04c9db8, State: Initialized, Role: FOLLOWER
I20260927 14:57:41.652526 9661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1719781fad924b3d9315c164f04c9db8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1719781fad924b3d9315c164f04c9db8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46111 } }
I20260927 14:57:41.653190 9661 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1719781fad924b3d9315c164f04c9db8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:57:41.653285 9661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1719781fad924b3d9315c164f04c9db8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:57:41.653393 9661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1719781fad924b3d9315c164f04c9db8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:41.654457 9661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1719781fad924b3d9315c164f04c9db8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1719781fad924b3d9315c164f04c9db8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46111 } }
I20260927 14:57:41.654831 9661 leader_election.cc:302] T 00000000000000000000000000000000 P 1719781fad924b3d9315c164f04c9db8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1719781fad924b3d9315c164f04c9db8; no voters:
I20260927 14:57:41.655159 9661 leader_election.cc:288] T 00000000000000000000000000000000 P 1719781fad924b3d9315c164f04c9db8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:57:41.655227 9666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1719781fad924b3d9315c164f04c9db8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:57:41.655426 9666 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1719781fad924b3d9315c164f04c9db8 [term 1 LEADER]: Becoming Leader. State: Replica: 1719781fad924b3d9315c164f04c9db8, State: Running, Role: LEADER
I20260927 14:57:41.655746 9666 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1719781fad924b3d9315c164f04c9db8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1719781fad924b3d9315c164f04c9db8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46111 } }
I20260927 14:57:41.656343 9661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1719781fad924b3d9315c164f04c9db8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:57:41.657306 9667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1719781fad924b3d9315c164f04c9db8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1719781fad924b3d9315c164f04c9db8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1719781fad924b3d9315c164f04c9db8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46111 } } }
I20260927 14:57:41.657305 9668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1719781fad924b3d9315c164f04c9db8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1719781fad924b3d9315c164f04c9db8. Latest consensus state: current_term: 1 leader_uuid: "1719781fad924b3d9315c164f04c9db8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1719781fad924b3d9315c164f04c9db8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46111 } } }
I20260927 14:57:41.657588 9668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1719781fad924b3d9315c164f04c9db8 [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:41.658047 9667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1719781fad924b3d9315c164f04c9db8 [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:41.658331 9675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:57:41.661311 9675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:57:41.666694 9675 catalog_manager.cc:1347] Generated new cluster ID: 73267daed76648a48a6e7f9f9b57a173
I20260927 14:57:41.666822 9675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:57:41.676867 9675 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:57:41.678236 9675 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:57:41.686192 9675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1719781fad924b3d9315c164f04c9db8: Generated new TSK 0
I20260927 14:57:41.687000 9675 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:57:41.694123 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46111
--builtin_ntp_servers=127.0.98.20:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 14:57:41.807744 9685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:41.808053 9685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:41.808092 9685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:41.812422 9685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:41.812497 9685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:41.812558 9685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:57:41.817795 9685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:41.819020 9685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:41.820478 9685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 14:57:41.830775 9685 server_base.cc:1034] running on GCE node
W20260927 14:57:41.830638 9693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:41.830636 9690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:41.830649 9691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:41.831380 9685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:41.832080 9685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:41.833276 9685 hybrid_clock.cc:648] HybridClock initialized: now 1790521061833255 us; error 38 us; skew 500 ppm
I20260927 14:57:41.835299 9685 webserver.cc:460] Webserver started at http://127.0.98.1:43775/ using document root <none> and password file <none>
I20260927 14:57:41.835875 9685 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:41.835947 9685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:41.836163 9685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:41.838291 9685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/data/instance:
uuid: "b28ee31cfec7430597a141e016797c18"
format_stamp: "Formatted at 2026-09-27 14:57:41 on dist-test-slave-pz0h"
server_key: "a157546a81dd3064d9f9ff8ee2e9a14a"
server_key_iv: "0dc982252cc5a7676705f25b18cebe15"
server_key_version: "encryptionkey@0"
I20260927 14:57:41.838915 9685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b28ee31cfec7430597a141e016797c18"
format_stamp: "Formatted at 2026-09-27 14:57:41 on dist-test-slave-pz0h"
server_key: "a157546a81dd3064d9f9ff8ee2e9a14a"
server_key_iv: "0dc982252cc5a7676705f25b18cebe15"
server_key_version: "encryptionkey@0"
I20260927 14:57:41.842859 9685 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 14:57:41.845691 9699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:41.847052 9685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:57:41.847182 9685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "b28ee31cfec7430597a141e016797c18"
format_stamp: "Formatted at 2026-09-27 14:57:41 on dist-test-slave-pz0h"
server_key: "a157546a81dd3064d9f9ff8ee2e9a14a"
server_key_iv: "0dc982252cc5a7676705f25b18cebe15"
server_key_version: "encryptionkey@0"
I20260927 14:57:41.847317 9685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:41.872972 9685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:41.873725 9685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:41.873905 9685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:41.874514 9685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:41.875551 9685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:41.875600 9685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:41.875655 9685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:41.875684 9685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:41.884748 9685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33567
I20260927 14:57:41.884780 9812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33567 every 8 connection(s)
I20260927 14:57:41.885753 9685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:57:41.890774 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 9685
I20260927 14:57:41.890916 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:57:41.891175 392 external_mini_cluster.cc:1428] Setting key 8b7d7e40abf71a4ef3d3d5a4c8c38b60
I20260927 14:57:41.893128 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46111
--builtin_ntp_servers=127.0.98.20:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 14:57:41.896131 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:41.887624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47309 (local address 127.0.98.62:46111)
0927 14:57:41.887761 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:41.887764 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:41.888866 (+ 1102us) server_negotiation.cc:407] Connection header received
0927 14:57:41.890312 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:41.890315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:41.890368 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:41.890450 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:41.891575 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:41.892209 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:41.893561 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:41.893892 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:41.894002 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:41.895435 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:41.895439 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:41.895441 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:57:41.895455 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:41.895477 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:57:41.895485 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:41.895558 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:41.895937 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 14:57:41.896007 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 14:57:41.897279 9815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:41.888023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46111 (local address 127.0.98.1:47309)
0927 14:57:41.888691 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:41.888760 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:41.890053 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:41.890648 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:41.890656 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:41.890763 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:41.891416 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:41.891433 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:41.892353 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:41.892357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:41.893430 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:41.893443 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:41.894091 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:41.894095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:41.894285 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:41.895050 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:57:41.895052 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:41.895118 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:57:41.895222 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:41.895575 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:41.895584 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:41.895589 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 14:57:41.895933 (+ 344us) client_negotiation.cc:240] Negotiation successful
0927 14:57:41.896381 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":115,"threads_started":1}
I20260927 14:57:41.899395 9813 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46111
I20260927 14:57:41.899706 9813 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:41.900465 9813 heartbeater.cc:511] Master 127.0.98.62:46111 requested a full tablet report, sending...
I20260927 14:57:41.902902 9625 ts_manager.cc:194] Registered new tserver with Master: b28ee31cfec7430597a141e016797c18 (127.0.98.1:33567)
I20260927 14:57:41.904446 9625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:47309
W20260927 14:57:42.010509 9816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:42.010842 9816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:42.010898 9816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:42.015213 9816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:42.015295 9816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:42.015358 9816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:57:42.020598 9816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:42.021703 9816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:42.023130 9816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:42.033545 9824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:42.033566 9822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:42.033566 9821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:42.033818 9816 server_base.cc:1034] running on GCE node
I20260927 14:57:42.034518 9816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:42.035302 9816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:42.036511 9816 hybrid_clock.cc:648] HybridClock initialized: now 1790521062036468 us; error 57 us; skew 500 ppm
I20260927 14:57:42.038745 9816 webserver.cc:460] Webserver started at http://127.0.98.2:43493/ using document root <none> and password file <none>
I20260927 14:57:42.039410 9816 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:42.039481 9816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:42.039680 9816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:42.041838 9816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/data/instance:
uuid: "aa524c25a5a54663be81c356dcfd8ba2"
format_stamp: "Formatted at 2026-09-27 14:57:42 on dist-test-slave-pz0h"
server_key: "f6fd890e8d6940bb2d00549792627ff7"
server_key_iv: "48daddd050b30a753be281d19a2b6735"
server_key_version: "encryptionkey@0"
I20260927 14:57:42.042397 9816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "aa524c25a5a54663be81c356dcfd8ba2"
format_stamp: "Formatted at 2026-09-27 14:57:42 on dist-test-slave-pz0h"
server_key: "f6fd890e8d6940bb2d00549792627ff7"
server_key_iv: "48daddd050b30a753be281d19a2b6735"
server_key_version: "encryptionkey@0"
I20260927 14:57:42.046605 9816 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260927 14:57:42.049702 9830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:42.051252 9816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:57:42.051395 9816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "aa524c25a5a54663be81c356dcfd8ba2"
format_stamp: "Formatted at 2026-09-27 14:57:42 on dist-test-slave-pz0h"
server_key: "f6fd890e8d6940bb2d00549792627ff7"
server_key_iv: "48daddd050b30a753be281d19a2b6735"
server_key_version: "encryptionkey@0"
I20260927 14:57:42.051518 9816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:42.079789 9816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:42.080526 9816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:42.080711 9816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:42.081300 9816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:42.082268 9816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:42.082317 9816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:42.082371 9816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:42.082398 9816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:42.091333 9816 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39733
I20260927 14:57:42.091382 9943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39733 every 8 connection(s)
I20260927 14:57:42.092315 9816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:57:42.101807 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 9816
I20260927 14:57:42.101929 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:57:42.101926 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:42.094383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47683 (local address 127.0.98.62:46111)
0927 14:57:42.094561 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:42.094565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:42.095772 (+ 1207us) server_negotiation.cc:407] Connection header received
0927 14:57:42.097175 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:42.097179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:42.097245 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:42.097321 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:42.098263 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:42.098837 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:42.099977 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:42.100195 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:42.100286 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:42.101425 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:42.101427 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:42.101429 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:57:42.101443 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:42.101464 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:57:42.101472 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:42.101532 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:42.101679 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 14:57:42.101758 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 14:57:42.102249 392 external_mini_cluster.cc:1428] Setting key dcd7a324a7436a91072a7ebdb84855dd
I20260927 14:57:42.103022 9946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:42.094838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46111 (local address 127.0.98.2:47683)
0927 14:57:42.095592 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:42.095636 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:42.096952 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:42.097514 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:42.097522 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:42.097610 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:42.098115 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:42.098128 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:42.098951 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:42.098953 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:42.099849 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:42.099859 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:42.100353 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:42.100357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:42.100529 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:42.101122 (+ 593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:57:42.101124 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:42.101167 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:57:42.101265 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:42.101541 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:42.101549 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:42.101556 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 14:57:42.101670 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 14:57:42.102002 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":123,"threads_started":1}
I20260927 14:57:42.104154 9944 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46111
I20260927 14:57:42.104394 9944 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:42.104423 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46111
--builtin_ntp_servers=127.0.98.20:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 14:57:42.104941 9944 heartbeater.cc:511] Master 127.0.98.62:46111 requested a full tablet report, sending...
I20260927 14:57:42.106077 9625 ts_manager.cc:194] Registered new tserver with Master: aa524c25a5a54663be81c356dcfd8ba2 (127.0.98.2:39733)
I20260927 14:57:42.106706 9625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:47683
W20260927 14:57:42.219172 9947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:42.219467 9947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:42.219504 9947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:42.223749 9947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:42.223824 9947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:42.223883 9947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:57:42.228886 9947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:42.229996 9947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:42.231477 9947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:42.241091 9955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:42.241082 9952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:42.241082 9953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:42.241325 9947 server_base.cc:1034] running on GCE node
I20260927 14:57:42.241823 9947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:42.242488 9947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:42.243644 9947 hybrid_clock.cc:648] HybridClock initialized: now 1790521062243602 us; error 62 us; skew 500 ppm
I20260927 14:57:42.245584 9947 webserver.cc:460] Webserver started at http://127.0.98.3:32937/ using document root <none> and password file <none>
I20260927 14:57:42.246181 9947 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:42.246253 9947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:42.246459 9947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:42.248692 9947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/data/instance:
uuid: "3aec453e498745f0b824d32430bd3e43"
format_stamp: "Formatted at 2026-09-27 14:57:42 on dist-test-slave-pz0h"
server_key: "503dab2ae286bbf3406acb0da7594008"
server_key_iv: "22f8e9917574a2a1ce600f4ad35a9630"
server_key_version: "encryptionkey@0"
I20260927 14:57:42.249303 9947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3aec453e498745f0b824d32430bd3e43"
format_stamp: "Formatted at 2026-09-27 14:57:42 on dist-test-slave-pz0h"
server_key: "503dab2ae286bbf3406acb0da7594008"
server_key_iv: "22f8e9917574a2a1ce600f4ad35a9630"
server_key_version: "encryptionkey@0"
I20260927 14:57:42.253358 9947 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 14:57:42.256165 9961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:42.257512 9947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 14:57:42.257632 9947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "3aec453e498745f0b824d32430bd3e43"
format_stamp: "Formatted at 2026-09-27 14:57:42 on dist-test-slave-pz0h"
server_key: "503dab2ae286bbf3406acb0da7594008"
server_key_iv: "22f8e9917574a2a1ce600f4ad35a9630"
server_key_version: "encryptionkey@0"
I20260927 14:57:42.257756 9947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:42.278946 9947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:42.279654 9947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:42.279832 9947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:42.280411 9947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:42.281395 9947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:42.281443 9947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:42.281498 9947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:42.281526 9947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:42.289848 9947 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44727
I20260927 14:57:42.289880 10074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44727 every 8 connection(s)
I20260927 14:57:42.290774 9947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:57:42.291169 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 9947
I20260927 14:57:42.291273 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:57:42.291543 392 external_mini_cluster.cc:1428] Setting key 7a178100c8ac91d96a40e1278d736a22
I20260927 14:57:42.300087 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:42.292665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55749 (local address 127.0.98.62:46111)
0927 14:57:42.292803 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:42.292806 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:42.293957 (+ 1151us) server_negotiation.cc:407] Connection header received
0927 14:57:42.295385 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:42.295389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:42.295442 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:42.295515 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:42.296426 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:42.296953 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:42.298092 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:42.298314 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:42.298403 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:42.299593 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:42.299597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:42.299599 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:57:42.299613 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:42.299633 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:57:42.299640 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:42.299712 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:42.299861 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 14:57:42.299931 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 14:57:42.300983 10077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:42.293111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46111 (local address 127.0.98.3:55749)
0927 14:57:42.293820 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:42.293862 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:42.295184 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:42.295694 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:42.295701 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:42.295791 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:42.296291 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:42.296304 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:42.297073 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:42.297076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:42.297967 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:42.297978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:42.298459 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:42.298464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:42.298622 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:42.299300 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:57:42.299302 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:42.299347 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:57:42.299421 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:42.299722 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:42.299732 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:42.299737 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 14:57:42.299854 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 14:57:42.300169 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":119,"threads_started":1}
I20260927 14:57:42.302117 10075 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46111
I20260927 14:57:42.302371 10075 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:42.303000 10075 heartbeater.cc:511] Master 127.0.98.62:46111 requested a full tablet report, sending...
I20260927 14:57:42.304184 9625 ts_manager.cc:194] Registered new tserver with Master: 3aec453e498745f0b824d32430bd3e43 (127.0.98.3:44727)
I20260927 14:57:42.304852 9625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:55749
I20260927 14:57:42.305428 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 14:57:42.308099 9664 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:42.307331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33276 (local address 127.0.98.62:46111)
0927 14:57:42.307443 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:42.307448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:42.307597 (+ 149us) server_negotiation.cc:407] Connection header received
0927 14:57:42.307694 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:42.307697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:42.307734 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:42.307813 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:42.307995 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33276: BlockingRecv error: recv got EOF from 127.0.0.1:33276 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 14:57:42.308270 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 9685
I20260927 14:57:42.313910 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 9816
I20260927 14:57:42.318910 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 9947
I20260927 14:57:42.323866 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 9594
2026-09-27T14:57:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (894 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T14:57:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:57:42Z Disabled control of system clock
I20260927 14:57:42.351356 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33901
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 14:57:42.466897 10086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:42.467182 10086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:42.467221 10086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:42.471324 10086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:57:42.471395 10086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:42.471408 10086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:42.471423 10086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:57:42.471437 10086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:57:42.476359 10086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33901
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:42.477497 10086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:42.478938 10086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:42.487221 10091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:42.487205 10094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:42.487385 10086 server_base.cc:1034] running on GCE node
W20260927 14:57:42.487205 10092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:42.488130 10086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:42.489269 10086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:42.490434 10086 hybrid_clock.cc:648] HybridClock initialized: now 1790521062490422 us; error 32 us; skew 500 ppm
I20260927 14:57:42.492350 10086 webserver.cc:460] Webserver started at http://127.0.98.62:37169/ using document root <none> and password file <none>
I20260927 14:57:42.492938 10086 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:42.493005 10086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:42.493206 10086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:42.495440 10086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "d36e3ad3c785445d8ae40f78e7481f54"
format_stamp: "Formatted at 2026-09-27 14:57:42 on dist-test-slave-pz0h"
server_key: "c42868be0813e8bec53d9f3e81d0fd54"
server_key_iv: "f5eb06fd081a6912bb37b2b36fa29eb1"
server_key_version: "encryptionkey@0"
I20260927 14:57:42.496021 10086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "d36e3ad3c785445d8ae40f78e7481f54"
format_stamp: "Formatted at 2026-09-27 14:57:42 on dist-test-slave-pz0h"
server_key: "c42868be0813e8bec53d9f3e81d0fd54"
server_key_iv: "f5eb06fd081a6912bb37b2b36fa29eb1"
server_key_version: "encryptionkey@0"
I20260927 14:57:42.499825 10086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 14:57:42.502473 10100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:42.503844 10086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 14:57:42.503970 10086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "d36e3ad3c785445d8ae40f78e7481f54"
format_stamp: "Formatted at 2026-09-27 14:57:42 on dist-test-slave-pz0h"
server_key: "c42868be0813e8bec53d9f3e81d0fd54"
server_key_iv: "f5eb06fd081a6912bb37b2b36fa29eb1"
server_key_version: "encryptionkey@0"
I20260927 14:57:42.504082 10086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:42.508760 10086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:42.509390 10086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:42.509555 10086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:42.516232 10086 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33901
I20260927 14:57:42.516250 10152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33901 every 8 connection(s)
I20260927 14:57:42.517184 10086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:57:42.520450 10153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:42.526194 10153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d36e3ad3c785445d8ae40f78e7481f54: Bootstrap starting.
I20260927 14:57:42.527468 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 10086
I20260927 14:57:42.527588 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 14:57:42.527860 392 external_mini_cluster.cc:1428] Setting key ee0242942239c294ef17b514abfad77e
I20260927 14:57:42.529181 10153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d36e3ad3c785445d8ae40f78e7481f54: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:42.529974 10153 log.cc:824] T 00000000000000000000000000000000 P d36e3ad3c785445d8ae40f78e7481f54: Log is configured to *not* fsync() on all Append() calls
I20260927 14:57:42.532384 10153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d36e3ad3c785445d8ae40f78e7481f54: No bootstrap required, opened a new log
I20260927 14:57:42.533535 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:42.529079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32954 (local address 127.0.98.62:33901)
0927 14:57:42.529733 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:42.529749 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:42.529790 (+ 41us) server_negotiation.cc:407] Connection header received
0927 14:57:42.530589 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:42.530617 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:42.531031 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:42.531361 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:42.532194 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:42.532199 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:42.532211 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:57:42.532222 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:42.532288 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:57:42.532307 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:42.532399 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:42.532541 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 14:57:42.532809 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":141,"threads_started":1}
I20260927 14:57:42.536375 10153 raft_consensus.cc:348] T 00000000000000000000000000000000 P d36e3ad3c785445d8ae40f78e7481f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36e3ad3c785445d8ae40f78e7481f54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33901 } }
I20260927 14:57:42.536620 10153 raft_consensus.cc:374] T 00000000000000000000000000000000 P d36e3ad3c785445d8ae40f78e7481f54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:42.536657 10153 raft_consensus.cc:729] T 00000000000000000000000000000000 P d36e3ad3c785445d8ae40f78e7481f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d36e3ad3c785445d8ae40f78e7481f54, State: Initialized, Role: FOLLOWER
I20260927 14:57:42.537109 10153 consensus_queue.cc:260] T 00000000000000000000000000000000 P d36e3ad3c785445d8ae40f78e7481f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36e3ad3c785445d8ae40f78e7481f54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33901 } }
I20260927 14:57:42.537241 10153 raft_consensus.cc:388] T 00000000000000000000000000000000 P d36e3ad3c785445d8ae40f78e7481f54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:57:42.537303 10153 raft_consensus.cc:482] T 00000000000000000000000000000000 P d36e3ad3c785445d8ae40f78e7481f54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:57:42.537402 10153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d36e3ad3c785445d8ae40f78e7481f54 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:42.538393 10153 raft_consensus.cc:504] T 00000000000000000000000000000000 P d36e3ad3c785445d8ae40f78e7481f54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36e3ad3c785445d8ae40f78e7481f54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33901 } }
I20260927 14:57:42.538760 10153 leader_election.cc:302] T 00000000000000000000000000000000 P d36e3ad3c785445d8ae40f78e7481f54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d36e3ad3c785445d8ae40f78e7481f54; no voters:
I20260927 14:57:42.539137 10153 leader_election.cc:288] T 00000000000000000000000000000000 P d36e3ad3c785445d8ae40f78e7481f54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:57:42.539325 10158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d36e3ad3c785445d8ae40f78e7481f54 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:57:42.539531 10158 raft_consensus.cc:686] T 00000000000000000000000000000000 P d36e3ad3c785445d8ae40f78e7481f54 [term 1 LEADER]: Becoming Leader. State: Replica: d36e3ad3c785445d8ae40f78e7481f54, State: Running, Role: LEADER
I20260927 14:57:42.539893 10158 consensus_queue.cc:237] T 00000000000000000000000000000000 P d36e3ad3c785445d8ae40f78e7481f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36e3ad3c785445d8ae40f78e7481f54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33901 } }
I20260927 14:57:42.540467 10153 sys_catalog.cc:564] T 00000000000000000000000000000000 P d36e3ad3c785445d8ae40f78e7481f54 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:57:42.541447 10159 sys_catalog.cc:455] T 00000000000000000000000000000000 P d36e3ad3c785445d8ae40f78e7481f54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d36e3ad3c785445d8ae40f78e7481f54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36e3ad3c785445d8ae40f78e7481f54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33901 } } }
I20260927 14:57:42.541574 10159 sys_catalog.cc:458] T 00000000000000000000000000000000 P d36e3ad3c785445d8ae40f78e7481f54 [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:42.541851 10160 sys_catalog.cc:455] T 00000000000000000000000000000000 P d36e3ad3c785445d8ae40f78e7481f54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d36e3ad3c785445d8ae40f78e7481f54. Latest consensus state: current_term: 1 leader_uuid: "d36e3ad3c785445d8ae40f78e7481f54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36e3ad3c785445d8ae40f78e7481f54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33901 } } }
I20260927 14:57:42.541967 10160 sys_catalog.cc:458] T 00000000000000000000000000000000 P d36e3ad3c785445d8ae40f78e7481f54 [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:42.542160 10167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:57:42.545115 10167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:57:42.550706 10167 catalog_manager.cc:1347] Generated new cluster ID: 2759e87eeefd47f48bd6f81d75224b51
I20260927 14:57:42.550781 10167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:57:42.564275 10167 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:57:42.564642 10167 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:57:42.569398 10167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d36e3ad3c785445d8ae40f78e7481f54: Generated new TSK 0
I20260927 14:57:42.570111 10167 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:57:42.579758 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33901
--builtin_ntp_servers=127.0.98.20:44273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 14:57:42.692214 10177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:42.692505 10177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:42.692548 10177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:42.696745 10177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:42.696815 10177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:42.696874 10177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:57:42.701793 10177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:42.703014 10177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:42.704416 10177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:42.714195 10183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:42.714175 10182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:42.714175 10185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:42.714273 10177 server_base.cc:1034] running on GCE node
I20260927 14:57:42.714958 10177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:42.715631 10177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:42.716806 10177 hybrid_clock.cc:648] HybridClock initialized: now 1790521062716779 us; error 45 us; skew 500 ppm
I20260927 14:57:42.718642 10177 webserver.cc:460] Webserver started at http://127.0.98.1:33653/ using document root <none> and password file <none>
I20260927 14:57:42.719251 10177 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:42.719321 10177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:42.719521 10177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:42.721654 10177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/data/instance:
uuid: "0d6e2353c0554d44961ffbf786fadb02"
format_stamp: "Formatted at 2026-09-27 14:57:42 on dist-test-slave-pz0h"
server_key: "4e31dedeff82bb50b6c33c94c1648170"
server_key_iv: "b7989173dc5232f00cea7e4ba20987c3"
server_key_version: "encryptionkey@0"
I20260927 14:57:42.722258 10177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0d6e2353c0554d44961ffbf786fadb02"
format_stamp: "Formatted at 2026-09-27 14:57:42 on dist-test-slave-pz0h"
server_key: "4e31dedeff82bb50b6c33c94c1648170"
server_key_iv: "b7989173dc5232f00cea7e4ba20987c3"
server_key_version: "encryptionkey@0"
I20260927 14:57:42.726298 10177 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 14:57:42.728955 10191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:42.730283 10177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:57:42.730413 10177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "0d6e2353c0554d44961ffbf786fadb02"
format_stamp: "Formatted at 2026-09-27 14:57:42 on dist-test-slave-pz0h"
server_key: "4e31dedeff82bb50b6c33c94c1648170"
server_key_iv: "b7989173dc5232f00cea7e4ba20987c3"
server_key_version: "encryptionkey@0"
I20260927 14:57:42.730528 10177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:42.735168 10177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:42.735813 10177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:42.735973 10177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:42.736507 10177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:42.737433 10177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:42.737480 10177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:42.737532 10177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:42.737557 10177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:42.745939 10177 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46755
I20260927 14:57:42.745983 10304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46755 every 8 connection(s)
I20260927 14:57:42.746913 10177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:57:42.753152 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:42.748711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38099 (local address 127.0.98.62:33901)
0927 14:57:42.748852 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:42.748856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:42.749963 (+ 1107us) server_negotiation.cc:407] Connection header received
0927 14:57:42.751374 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:42.751377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:42.751428 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:42.751503 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:42.752665 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:42.752670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:42.752686 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:57:42.752701 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:42.752723 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:57:42.752732 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:42.752820 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:42.752953 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 14:57:42.753004 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 14:57:42.754091 10307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:42.749112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33901 (local address 127.0.98.1:38099)
0927 14:57:42.749805 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:42.749848 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:42.751142 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:42.751684 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:42.751691 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:42.751789 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:42.752359 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:57:42.752360 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:42.752408 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:57:42.752488 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:42.752813 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:42.752821 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:42.752828 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 14:57:42.752998 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 14:57:42.753285 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":111,"threads_started":1}
I20260927 14:57:42.755460 10305 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33901
I20260927 14:57:42.755716 10305 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:42.756063 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 10177
I20260927 14:57:42.756161 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:57:42.756166 10305 heartbeater.cc:511] Master 127.0.98.62:33901 requested a full tablet report, sending...
I20260927 14:57:42.756429 392 external_mini_cluster.cc:1428] Setting key 641bf4f4d5a8917a9ce916beeb4eab5a
I20260927 14:57:42.757887 10116 ts_manager.cc:194] Registered new tserver with Master: 0d6e2353c0554d44961ffbf786fadb02 (127.0.98.1:46755)
I20260927 14:57:42.758838 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33901
--builtin_ntp_servers=127.0.98.20:44273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 14:57:42.871934 10308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:42.872243 10308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:42.872284 10308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:42.876631 10308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:42.876711 10308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:42.876772 10308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:57:42.881804 10308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:42.883028 10308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:42.884450 10308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:42.893906 10316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:42.893931 10313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:42.893924 10314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:42.894294 10308 server_base.cc:1034] running on GCE node
I20260927 14:57:42.894696 10308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:42.895422 10308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:42.896617 10308 hybrid_clock.cc:648] HybridClock initialized: now 1790521062896589 us; error 41 us; skew 500 ppm
I20260927 14:57:42.898890 10308 webserver.cc:460] Webserver started at http://127.0.98.2:32971/ using document root <none> and password file <none>
I20260927 14:57:42.899555 10308 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:42.899636 10308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:42.899964 10308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:42.902251 10308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/data/instance:
uuid: "af8fa8923fcf436bb76cbe3f53921828"
format_stamp: "Formatted at 2026-09-27 14:57:42 on dist-test-slave-pz0h"
server_key: "351355520115ddcf586faac12f07e711"
server_key_iv: "8325e177056f8f9ecb481516847f3540"
server_key_version: "encryptionkey@0"
I20260927 14:57:42.902935 10308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "af8fa8923fcf436bb76cbe3f53921828"
format_stamp: "Formatted at 2026-09-27 14:57:42 on dist-test-slave-pz0h"
server_key: "351355520115ddcf586faac12f07e711"
server_key_iv: "8325e177056f8f9ecb481516847f3540"
server_key_version: "encryptionkey@0"
I20260927 14:57:42.907042 10308 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 14:57:42.910456 10322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:42.911980 10308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:57:42.912130 10308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "af8fa8923fcf436bb76cbe3f53921828"
format_stamp: "Formatted at 2026-09-27 14:57:42 on dist-test-slave-pz0h"
server_key: "351355520115ddcf586faac12f07e711"
server_key_iv: "8325e177056f8f9ecb481516847f3540"
server_key_version: "encryptionkey@0"
I20260927 14:57:42.912261 10308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:42.917177 10308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:42.917862 10308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:42.918040 10308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:42.918627 10308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:42.919654 10308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:42.919703 10308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:42.919755 10308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:42.919782 10308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:42.928162 10308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35277
I20260927 14:57:42.928187 10435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35277 every 8 connection(s)
I20260927 14:57:42.929100 10308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:57:42.935415 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:42.930977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33129 (local address 127.0.98.62:33901)
0927 14:57:42.931116 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:42.931120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:42.932181 (+ 1061us) server_negotiation.cc:407] Connection header received
0927 14:57:42.933566 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:42.933569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:42.933613 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:42.933694 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:42.935031 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:42.935034 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:42.935036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:57:42.935049 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:42.935069 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:57:42.935077 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:42.935165 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:42.935260 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 14:57:42.935299 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:57:42.935885 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 10308
I20260927 14:57:42.935992 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:57:42.936287 392 external_mini_cluster.cc:1428] Setting key 1f397f782b3ff7e5724580eb052dcd3b
I20260927 14:57:42.936568 10438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:42.931378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33901 (local address 127.0.98.2:33129)
0927 14:57:42.932030 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:42.932071 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:42.933280 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:42.933908 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:42.933916 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:42.934018 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:42.934713 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:57:42.934715 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:42.934759 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:57:42.934878 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:42.935150 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:42.935155 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:42.935159 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 14:57:42.935294 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 14:57:42.935508 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":110,"threads_started":1}
I20260927 14:57:42.937726 10436 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33901
I20260927 14:57:42.937969 10436 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:42.938370 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33901
--builtin_ntp_servers=127.0.98.20:44273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 14:57:42.938391 10436 heartbeater.cc:511] Master 127.0.98.62:33901 requested a full tablet report, sending...
I20260927 14:57:42.939486 10116 ts_manager.cc:194] Registered new tserver with Master: af8fa8923fcf436bb76cbe3f53921828 (127.0.98.2:35277)
W20260927 14:57:43.050222 10439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:43.050532 10439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:43.050588 10439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:43.054924 10439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:43.054999 10439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:43.055081 10439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:57:43.060209 10439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:43.061348 10439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:43.062763 10439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:43.072844 10447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:43.072844 10445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:43.072891 10439 server_base.cc:1034] running on GCE node
W20260927 14:57:43.072849 10444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:43.073493 10439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:43.074162 10439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:43.075306 10439 hybrid_clock.cc:648] HybridClock initialized: now 1790521063075278 us; error 45 us; skew 500 ppm
I20260927 14:57:43.077159 10439 webserver.cc:460] Webserver started at http://127.0.98.3:45205/ using document root <none> and password file <none>
I20260927 14:57:43.077718 10439 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:43.077785 10439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:43.077986 10439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:43.080190 10439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/data/instance:
uuid: "4733c487369e418db7c707f5d9cb25a7"
format_stamp: "Formatted at 2026-09-27 14:57:43 on dist-test-slave-pz0h"
server_key: "8f291534114d3ac37ffb3a3b05e0c3a0"
server_key_iv: "aed6da04940fdc805ce5c01172ecf642"
server_key_version: "encryptionkey@0"
I20260927 14:57:43.080778 10439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4733c487369e418db7c707f5d9cb25a7"
format_stamp: "Formatted at 2026-09-27 14:57:43 on dist-test-slave-pz0h"
server_key: "8f291534114d3ac37ffb3a3b05e0c3a0"
server_key_iv: "aed6da04940fdc805ce5c01172ecf642"
server_key_version: "encryptionkey@0"
I20260927 14:57:43.084755 10439 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 14:57:43.087381 10453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:43.088778 10439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:57:43.088927 10439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "4733c487369e418db7c707f5d9cb25a7"
format_stamp: "Formatted at 2026-09-27 14:57:43 on dist-test-slave-pz0h"
server_key: "8f291534114d3ac37ffb3a3b05e0c3a0"
server_key_iv: "aed6da04940fdc805ce5c01172ecf642"
server_key_version: "encryptionkey@0"
I20260927 14:57:43.089048 10439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:43.093967 10439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:43.094611 10439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:43.094777 10439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:43.095363 10439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:43.096293 10439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:43.096340 10439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:43.096391 10439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:43.096418 10439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:43.104822 10439 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42513
I20260927 14:57:43.104842 10566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42513 every 8 connection(s)
I20260927 14:57:43.105738 10439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:57:43.111946 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:43.107698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43223 (local address 127.0.98.62:33901)
0927 14:57:43.107862 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:43.107873 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:43.109084 (+ 1211us) server_negotiation.cc:407] Connection header received
0927 14:57:43.110341 (+ 1257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:43.110343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:43.110383 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:43.110459 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:43.111554 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:43.111558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:43.111560 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 14:57:43.111573 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:43.111595 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 14:57:43.111603 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:43.111692 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:43.111788 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 14:57:43.111826 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 14:57:43.112797 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:43.108155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33901 (local address 127.0.98.3:43223)
0927 14:57:43.108933 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:43.108977 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:43.110146 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:43.110610 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:43.110617 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:43.110721 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:43.111290 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 14:57:43.111292 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:43.111332 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 14:57:43.111402 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:43.111678 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:43.111682 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:43.111686 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 14:57:43.111823 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 14:57:43.112021 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":142,"threads_started":1}
I20260927 14:57:43.113879 10567 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33901
I20260927 14:57:43.114089 10567 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:43.114482 10567 heartbeater.cc:511] Master 127.0.98.62:33901 requested a full tablet report, sending...
I20260927 14:57:43.114672 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 10439
I20260927 14:57:43.114773 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790521004237705-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:57:43.115089 392 external_mini_cluster.cc:1428] Setting key a5033f1e3b6710e955d110112fcae98a
I20260927 14:57:43.115768 10116 ts_manager.cc:194] Registered new tserver with Master: 4733c487369e418db7c707f5d9cb25a7 (127.0.98.3:42513)
I20260927 14:57:43.116963 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 14:57:43.119675 10156 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:57:43.118780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32958 (local address 127.0.98.62:33901)
0927 14:57:43.118945 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:43.118949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:43.119070 (+ 121us) server_negotiation.cc:407] Connection header received
0927 14:57:43.119206 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:43.119209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:43.119254 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:43.119321 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:43.119543 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32958: BlockingRecv error: recv got EOF from 127.0.0.1:32958 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 14:57:43.119789 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 10177
I20260927 14:57:43.124789 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 10308
I20260927 14:57:43.129385 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 10439
I20260927 14:57:43.134143 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 10086
2026-09-27T14:57:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (810 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:57:43 dist-test-slave-pz0h krb5kdc[10577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:57:43 dist-test-slave-pz0h krb5kdc[10577](info): set up 2 sockets
Sep 27 14:57:43 dist-test-slave-pz0h krb5kdc[10577](info): commencing operation
krb5kdc: starting...
W20260927 14:57:44.087453 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 14:57:44 dist-test-slave-pz0h krb5kdc[10577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:57:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:57:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:57:44.209883 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42451
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:44.322181 10590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:44.322506 10590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:44.322564 10590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:44.326767 10590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:57:44.326881 10590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:44.326911 10590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:44.326932 10590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:57:44.326954 10590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:57:44.332609 10590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:44.333797 10590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:44.334833 10590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:44.343298 10595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:44.343298 10596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:44.343405 10590 server_base.cc:1034] running on GCE node
W20260927 14:57:44.343299 10598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:44.344121 10590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:44.345326 10590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:44.346524 10590 hybrid_clock.cc:648] HybridClock initialized: now 1790521064346500 us; error 39 us; skew 500 ppm
Sep 27 14:57:44 dist-test-slave-pz0h krb5kdc[10577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:44.349494 10590 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:57:44.350621 10590 webserver.cc:460] Webserver started at http://127.0.98.62:44847/ using document root <none> and password file <none>
I20260927 14:57:44.351317 10590 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:44.351397 10590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:44.351619 10590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:44.353695 10590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "0ab964e2dc66430ca699a1fd13706354"
format_stamp: "Formatted at 2026-09-27 14:57:44 on dist-test-slave-pz0h"
server_key: "db0d0f069ee0278379c40c8d5748da8d"
server_key_iv: "cdc248ffd053db7cea7a21197d0ca134"
server_key_version: "encryptionkey@0"
I20260927 14:57:44.354233 10590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "0ab964e2dc66430ca699a1fd13706354"
format_stamp: "Formatted at 2026-09-27 14:57:44 on dist-test-slave-pz0h"
server_key: "db0d0f069ee0278379c40c8d5748da8d"
server_key_iv: "cdc248ffd053db7cea7a21197d0ca134"
server_key_version: "encryptionkey@0"
I20260927 14:57:44.357882 10590 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 14:57:44.360360 10605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:44.361630 10590 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:57:44.361747 10590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "0ab964e2dc66430ca699a1fd13706354"
format_stamp: "Formatted at 2026-09-27 14:57:44 on dist-test-slave-pz0h"
server_key: "db0d0f069ee0278379c40c8d5748da8d"
server_key_iv: "cdc248ffd053db7cea7a21197d0ca134"
server_key_version: "encryptionkey@0"
I20260927 14:57:44.361850 10590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:44.378295 10590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:44.381538 10590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:44.381747 10590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:44.388844 10590 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42451
I20260927 14:57:44.389149 10657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42451 every 8 connection(s)
I20260927 14:57:44.389837 10590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:57:44.392627 10658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:44.396471 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 10590
I20260927 14:57:44.396579 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 14:57:44.396808 392 external_mini_cluster.cc:1428] Setting key f127252cb4ca0da953ee26a77d62f0a7
I20260927 14:57:44.398355 10658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ab964e2dc66430ca699a1fd13706354: Bootstrap starting.
I20260927 14:57:44.400866 10658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ab964e2dc66430ca699a1fd13706354: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:44.401623 10658 log.cc:824] T 00000000000000000000000000000000 P 0ab964e2dc66430ca699a1fd13706354: Log is configured to *not* fsync() on all Append() calls
I20260927 14:57:44.403936 10658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ab964e2dc66430ca699a1fd13706354: No bootstrap required, opened a new log
Sep 27 14:57:44 dist-test-slave-pz0h krb5kdc[10577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:57:44.408586 10658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ab964e2dc66430ca699a1fd13706354 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab964e2dc66430ca699a1fd13706354" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42451 } }
I20260927 14:57:44.408932 10658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ab964e2dc66430ca699a1fd13706354 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:44.408984 10658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ab964e2dc66430ca699a1fd13706354 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ab964e2dc66430ca699a1fd13706354, State: Initialized, Role: FOLLOWER
I20260927 14:57:44.409495 10658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ab964e2dc66430ca699a1fd13706354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab964e2dc66430ca699a1fd13706354" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42451 } }
I20260927 14:57:44.409651 10658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ab964e2dc66430ca699a1fd13706354 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:57:44.409726 10658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ab964e2dc66430ca699a1fd13706354 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:57:44.409827 10658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ab964e2dc66430ca699a1fd13706354 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:44.411127 10658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ab964e2dc66430ca699a1fd13706354 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab964e2dc66430ca699a1fd13706354" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42451 } }
I20260927 14:57:44.411557 10658 leader_election.cc:302] T 00000000000000000000000000000000 P 0ab964e2dc66430ca699a1fd13706354 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ab964e2dc66430ca699a1fd13706354; no voters:
I20260927 14:57:44.411935 10658 leader_election.cc:288] T 00000000000000000000000000000000 P 0ab964e2dc66430ca699a1fd13706354 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:57:44.412038 10663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ab964e2dc66430ca699a1fd13706354 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:57:44.412256 10663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ab964e2dc66430ca699a1fd13706354 [term 1 LEADER]: Becoming Leader. State: Replica: 0ab964e2dc66430ca699a1fd13706354, State: Running, Role: LEADER
I20260927 14:57:44.412612 10663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ab964e2dc66430ca699a1fd13706354 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab964e2dc66430ca699a1fd13706354" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42451 } }
I20260927 14:57:44.412678 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:44.398168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38728 (local address 127.0.98.62:42451)
0927 14:57:44.398556 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:44.398571 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:44.398605 (+ 34us) server_negotiation.cc:407] Connection header received
0927 14:57:44.399274 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:44.399292 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:44.399616 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:44.399913 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:44.400789 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:44.401879 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:44.402944 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:44.403420 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:44.403540 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:44.406916 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:44.406927 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:44.406953 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:44.407004 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:44.409101 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:44.409666 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:44.409674 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:44.409680 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:44.409840 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:44.410111 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:44.410114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:44.410116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:44.410685 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:44.410926 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:44.411202 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 14:57:44.411549 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":107,"threads_started":1}
I20260927 14:57:44.413308 10658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ab964e2dc66430ca699a1fd13706354 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:57:44.414302 10665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab964e2dc66430ca699a1fd13706354 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ab964e2dc66430ca699a1fd13706354. Latest consensus state: current_term: 1 leader_uuid: "0ab964e2dc66430ca699a1fd13706354" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab964e2dc66430ca699a1fd13706354" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42451 } } }
I20260927 14:57:44.414428 10665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab964e2dc66430ca699a1fd13706354 [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:44.414835 10664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab964e2dc66430ca699a1fd13706354 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ab964e2dc66430ca699a1fd13706354" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab964e2dc66430ca699a1fd13706354" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42451 } } }
I20260927 14:57:44.414923 10664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab964e2dc66430ca699a1fd13706354 [sys.catalog]: This master's current role is: LEADER
I20260927 14:57:44.415076 10672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:57:44.418066 10672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:57:44.423679 10672 catalog_manager.cc:1347] Generated new cluster ID: b9a46e5da731443ab0b6abe1d2139b88
I20260927 14:57:44.423763 10672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:57:44.435731 10672 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:57:44.436857 10672 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:57:44.444800 10672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ab964e2dc66430ca699a1fd13706354: Generated new TSK 0
I20260927 14:57:44.445559 10672 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:57:44.473348 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42451
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:44.587363 10684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:44.587664 10684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:44.587710 10684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:44.591964 10684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:44.592046 10684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:44.592108 10684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:57:44.597874 10684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:44.599352 10684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:44.600553 10684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:44.610615 10689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:44.610615 10692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:44.610677 10684 server_base.cc:1034] running on GCE node
W20260927 14:57:44.610615 10690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:44.611352 10684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:44.612057 10684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:44.613245 10684 hybrid_clock.cc:648] HybridClock initialized: now 1790521064613225 us; error 38 us; skew 500 ppm
Sep 27 14:57:44 dist-test-slave-pz0h krb5kdc[10577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:44.616371 10684 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:57:44.617359 10684 webserver.cc:460] Webserver started at http://127.0.98.1:38869/ using document root <none> and password file <none>
I20260927 14:57:44.617913 10684 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:44.617980 10684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:44.618175 10684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:44.620445 10684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-0/data/instance:
uuid: "080b23a41a12406aafece26788b802ff"
format_stamp: "Formatted at 2026-09-27 14:57:44 on dist-test-slave-pz0h"
server_key: "8eeab908f66996521e69a0b1b1c332a4"
server_key_iv: "caa5183cbfc1459356d497d0b571d1dc"
server_key_version: "encryptionkey@0"
I20260927 14:57:44.621032 10684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "080b23a41a12406aafece26788b802ff"
format_stamp: "Formatted at 2026-09-27 14:57:44 on dist-test-slave-pz0h"
server_key: "8eeab908f66996521e69a0b1b1c332a4"
server_key_iv: "caa5183cbfc1459356d497d0b571d1dc"
server_key_version: "encryptionkey@0"
I20260927 14:57:44.624970 10684 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 14:57:44.627698 10699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:44.629081 10684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:57:44.629210 10684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "080b23a41a12406aafece26788b802ff"
format_stamp: "Formatted at 2026-09-27 14:57:44 on dist-test-slave-pz0h"
server_key: "8eeab908f66996521e69a0b1b1c332a4"
server_key_iv: "caa5183cbfc1459356d497d0b571d1dc"
server_key_version: "encryptionkey@0"
I20260927 14:57:44.629318 10684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:44.651787 10684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:44.655150 10684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:44.655346 10684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:44.655966 10684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:44.656852 10684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:44.656903 10684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:44.656960 10684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:44.656987 10684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:44.665961 10684 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35781
I20260927 14:57:44.665984 10812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35781 every 8 connection(s)
I20260927 14:57:44.666953 10684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:57:44.669966 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 10684
I20260927 14:57:44.670073 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:57:44.670334 392 external_mini_cluster.cc:1428] Setting key a4c09322dc43bc7834438a9b9be9188e
Sep 27 14:57:44 dist-test-slave-pz0h krb5kdc[10577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 14:57:44.681572 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:44.668757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43573 (local address 127.0.98.62:42451)
0927 14:57:44.668905 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:44.668908 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:44.670046 (+ 1138us) server_negotiation.cc:407] Connection header received
0927 14:57:44.671039 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:44.671043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:44.671099 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:44.671174 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:44.672469 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:44.673082 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:44.674463 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:44.674719 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:44.674853 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:44.677878 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:44.677883 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:44.677886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:44.677919 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:44.679472 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:44.680016 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:44.680019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:44.680020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:44.680097 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:44.680475 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:44.680478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:44.680479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:44.680676 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:44.680764 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:44.681305 (+ 541us) server_negotiation.cc:299] Negotiation successful
0927 14:57:44.681445 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 14:57:44.682463 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:44.669151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.1:43573)
0927 14:57:44.669883 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:44.669929 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:44.670848 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:44.671373 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:44.671381 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:44.671820 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:44.672328 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:44.672343 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:44.673238 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:44.673241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:44.674301 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:44.674315 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:44.674907 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:44.674911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:44.675123 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:44.675818 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:44.675843 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:44.677650 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:44.679594 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:44.679599 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:44.679614 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:44.679920 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:44.680218 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:44.680221 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:44.680222 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:44.680397 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:44.680773 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:44.680779 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:44.681046 (+ 267us) client_negotiation.cc:769] Sending connection context
0927 14:57:44.681209 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 14:57:44.681498 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":131,"threads_started":1}
I20260927 14:57:44.683720 10813 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42451
I20260927 14:57:44.683964 10813 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:44.684453 10813 heartbeater.cc:511] Master 127.0.98.62:42451 requested a full tablet report, sending...
I20260927 14:57:44.685931 10622 ts_manager.cc:194] Registered new tserver with Master: 080b23a41a12406aafece26788b802ff (127.0.98.1:35781)
I20260927 14:57:44.687104 10622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:57:44.697490 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42451
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:44.809877 10818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:44.810159 10818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:44.810204 10818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:44.814483 10818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:44.814551 10818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:44.814622 10818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:57:44.820143 10818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:44.821187 10818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:44.822173 10818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:57:44.831625 10823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:44.831777 10818 server_base.cc:1034] running on GCE node
W20260927 14:57:44.831938 10824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:44.831977 10826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:44.832422 10818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:44.833159 10818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:44.834342 10818 hybrid_clock.cc:648] HybridClock initialized: now 1790521064834320 us; error 34 us; skew 500 ppm
Sep 27 14:57:44 dist-test-slave-pz0h krb5kdc[10577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:44.837265 10818 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:57:44.838253 10818 webserver.cc:460] Webserver started at http://127.0.98.2:36213/ using document root <none> and password file <none>
I20260927 14:57:44.838788 10818 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:44.838891 10818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:44.839067 10818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:44.841091 10818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-1/data/instance:
uuid: "d8be40d5917041069d71a3419bd68958"
format_stamp: "Formatted at 2026-09-27 14:57:44 on dist-test-slave-pz0h"
server_key: "d0a4b6df89963b2a51773f06cdbb2798"
server_key_iv: "2eda697c90c8ce5979a6ac3c82508115"
server_key_version: "encryptionkey@0"
I20260927 14:57:44.841624 10818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d8be40d5917041069d71a3419bd68958"
format_stamp: "Formatted at 2026-09-27 14:57:44 on dist-test-slave-pz0h"
server_key: "d0a4b6df89963b2a51773f06cdbb2798"
server_key_iv: "2eda697c90c8ce5979a6ac3c82508115"
server_key_version: "encryptionkey@0"
I20260927 14:57:44.845510 10818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 14:57:44.847920 10833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:44.849125 10818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 14:57:44.849239 10818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "d8be40d5917041069d71a3419bd68958"
format_stamp: "Formatted at 2026-09-27 14:57:44 on dist-test-slave-pz0h"
server_key: "d0a4b6df89963b2a51773f06cdbb2798"
server_key_iv: "2eda697c90c8ce5979a6ac3c82508115"
server_key_version: "encryptionkey@0"
I20260927 14:57:44.849349 10818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:44.873027 10818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:44.876216 10818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:44.876382 10818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:44.876995 10818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:44.877880 10818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:44.877928 10818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:44.877969 10818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:44.877982 10818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:44.886947 10818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33481
I20260927 14:57:44.886984 10946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33481 every 8 connection(s)
I20260927 14:57:44.887921 10818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:57:44.894181 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 10818
I20260927 14:57:44.894315 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:57:44.894630 392 external_mini_cluster.cc:1428] Setting key fa8e9cf5a3bc11007b5d152ce7910db2
Sep 27 14:57:44 dist-test-slave-pz0h krb5kdc[10577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:57:44.902786 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:44.889956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43693 (local address 127.0.98.62:42451)
0927 14:57:44.890116 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:44.890120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:44.891235 (+ 1115us) server_negotiation.cc:407] Connection header received
0927 14:57:44.892167 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:44.892171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:44.892230 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:44.892306 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:44.893531 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:44.894071 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:44.895601 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:44.895865 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:44.895979 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:44.898747 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:44.898750 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:44.898752 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:44.898780 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:44.900175 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:44.900910 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:44.900914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:44.900915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:44.901015 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:44.901440 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:44.901442 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:44.901444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:44.901642 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:44.901715 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:44.902460 (+ 745us) server_negotiation.cc:299] Negotiation successful
0927 14:57:44.902614 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 14:57:44.904037 10949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:44.890365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.2:43693)
0927 14:57:44.891080 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:44.891125 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:44.891918 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:44.892472 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:44.892479 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:44.892867 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:44.893374 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:44.893389 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:44.894224 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:44.894228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:44.895452 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:44.895464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:44.896043 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:44.896047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:44.896234 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:44.896849 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:44.896866 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:44.898506 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:44.900338 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:44.900345 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:44.900367 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:44.900778 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:44.901116 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:44.901119 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:44.901121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:44.901353 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:44.901737 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:44.901745 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:44.902136 (+ 391us) client_negotiation.cc:769] Sending connection context
0927 14:57:44.902392 (+ 256us) client_negotiation.cc:240] Negotiation successful
0927 14:57:44.902847 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":124,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 14:57:44.905592 10947 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42451
I20260927 14:57:44.905906 10947 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:44.906450 10947 heartbeater.cc:511] Master 127.0.98.62:42451 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 14:57:44.907665 10622 ts_manager.cc:194] Registered new tserver with Master: d8be40d5917041069d71a3419bd68958 (127.0.98.2:33481)
I20260927 14:57:44.908399 10622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43693
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:57:44.922266 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42451
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:57:45.032629 10952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:57:45.033120 10952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:57:45.033182 10952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:57:45.037530 10952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:57:45.037599 10952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:57:45.037662 10952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:57:45.043210 10952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:57:45.044327 10952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:57:45.045341 10952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 14:57:45.056094 10952 server_base.cc:1034] running on GCE node
W20260927 14:57:45.056051 10957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:45.056051 10958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:57:45.056056 10960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:57:45.056820 10952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:57:45.057533 10952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:57:45.058732 10952 hybrid_clock.cc:648] HybridClock initialized: now 1790521065058698 us; error 49 us; skew 500 ppm
Sep 27 14:57:45 dist-test-slave-pz0h krb5kdc[10577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521065, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:57:45.061494 10952 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:57:45.062446 10952 webserver.cc:460] Webserver started at http://127.0.98.3:37621/ using document root <none> and password file <none>
I20260927 14:57:45.063058 10952 fs_manager.cc:359] Metadata directory not provided
I20260927 14:57:45.063125 10952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:57:45.063319 10952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:57:45.065492 10952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-2/data/instance:
uuid: "2a2ec4938faa483ca684479c870fcb09"
format_stamp: "Formatted at 2026-09-27 14:57:45 on dist-test-slave-pz0h"
server_key: "9e130e31818044eb7853fa16f18d7314"
server_key_iv: "ef799b499c5d19d33cd2b246818254fb"
server_key_version: "encryptionkey@0"
I20260927 14:57:45.066063 10952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2a2ec4938faa483ca684479c870fcb09"
format_stamp: "Formatted at 2026-09-27 14:57:45 on dist-test-slave-pz0h"
server_key: "9e130e31818044eb7853fa16f18d7314"
server_key_iv: "ef799b499c5d19d33cd2b246818254fb"
server_key_version: "encryptionkey@0"
I20260927 14:57:45.070500 10952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 14:57:45.073268 10967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:45.074576 10952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 14:57:45.074695 10952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "2a2ec4938faa483ca684479c870fcb09"
format_stamp: "Formatted at 2026-09-27 14:57:45 on dist-test-slave-pz0h"
server_key: "9e130e31818044eb7853fa16f18d7314"
server_key_iv: "ef799b499c5d19d33cd2b246818254fb"
server_key_version: "encryptionkey@0"
I20260927 14:57:45.074946 10952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:57:45.097651 10952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:57:45.100816 10952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:57:45.100996 10952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:57:45.101641 10952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:57:45.102540 10952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:57:45.102589 10952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:45.102653 10952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:57:45.102682 10952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:57:45.111383 10952 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36157
I20260927 14:57:45.111413 11080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36157 every 8 connection(s)
I20260927 14:57:45.112314 10952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:57:45.118832 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 10952
I20260927 14:57:45.118944 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:57:45.119196 392 external_mini_cluster.cc:1428] Setting key b439241babaa6ec15279d03cdba7593e
Sep 27 14:57:45 dist-test-slave-pz0h krb5kdc[10577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521065, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:57:45.125980 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:45.114158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37573 (local address 127.0.98.62:42451)
0927 14:57:45.114315 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:45.114319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:45.115607 (+ 1288us) server_negotiation.cc:407] Connection header received
0927 14:57:45.116522 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:45.116526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:45.116577 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:45.116659 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:45.117877 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:45.118412 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:45.119715 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:45.119945 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:45.120044 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:45.122474 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:45.122478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:45.122480 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:45.122505 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:45.123963 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:45.124468 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:45.124471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:45.124472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:45.124553 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:45.124885 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:45.124889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:45.124891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:45.125078 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:45.125167 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:45.125682 (+ 515us) server_negotiation.cc:299] Negotiation successful
0927 14:57:45.125814 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 14:57:45.126854 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:45.114637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.3:37573)
0927 14:57:45.115391 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:45.115436 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:45.116303 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:45.116882 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:45.116890 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:45.117264 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:57:45.117755 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:45.117769 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:45.118529 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:45.118532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:45.119596 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:45.119608 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:45.120101 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:45.120105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:45.120288 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:45.120876 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:57:45.120892 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:57:45.122305 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:57:45.124077 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:45.124084 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:45.124103 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:45.124369 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:45.124637 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:57:45.124641 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:57:45.124643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:57:45.124795 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:57:45.125181 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:57:45.125189 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:57:45.125450 (+ 261us) client_negotiation.cc:769] Sending connection context
0927 14:57:45.125626 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 14:57:45.125984 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":166,"threads_started":1}
I20260927 14:57:45.127960 11081 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42451
I20260927 14:57:45.128182 11081 heartbeater.cc:464] Registering TS with master...
I20260927 14:57:45.128640 11081 heartbeater.cc:511] Master 127.0.98.62:42451 requested a full tablet report, sending...
I20260927 14:57:45.129585 10622 ts_manager.cc:194] Registered new tserver with Master: 2a2ec4938faa483ca684479c870fcb09 (127.0.98.3:36157)
I20260927 14:57:45.130153 10622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37573
I20260927 14:57:45.132683 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:57:45.142109 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:45.134530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38744 (local address 127.0.98.62:42451)
0927 14:57:45.134654 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:45.134658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:45.134859 (+ 201us) server_negotiation.cc:407] Connection header received
0927 14:57:45.135000 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:45.135003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:45.135048 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:45.135113 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:57:45.135840 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:45.136367 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:45.137215 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:45.137438 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:45.137531 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:45.138972 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:57:45.138976 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:57:45.138978 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:57:45.139007 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:57:45.140152 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:45.140588 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:45.140590 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:45.140592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:45.140663 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:57:45.141167 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:57:45.141170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:57:45.141171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:57:45.141435 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:57:45.141556 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:45.141819 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 14:57:45.141945 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 14:57:45.145638 10622 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:57:45.148967 10622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:57:45.163679 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:45.158114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58834 (local address 127.0.98.2:33481)
0927 14:57:45.158440 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:45.158453 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:45.158477 (+ 24us) server_negotiation.cc:407] Connection header received
0927 14:57:45.158683 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:45.158689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:45.158948 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:45.159476 (+ 528us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:45.159638 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:45.160713 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:45.162092 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:45.163009 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:45.163104 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:45.163272 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:45.163414 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 14:57:45.163515 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":116,"threads_started":1}
I20260927 14:57:45.163669 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:45.157943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33481 (local address 127.0.0.1:58834)
0927 14:57:45.158303 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:45.158373 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:45.158507 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:45.159126 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:45.159132 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:45.159157 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:45.159305 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:45.159312 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:45.160822 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:45.160825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:45.161966 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:45.161976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:45.163123 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:45.163126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:45.163235 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:45.163271 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 14:57:45.163382 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 14:57:45.163516 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":55,"thread_start_us":98,"threads_started":1}
I20260927 14:57:45.164389 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:45.158686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35781 (local address 127.0.0.1:53586)
0927 14:57:45.158924 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:45.158936 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:45.159016 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:45.160441 (+ 1425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:45.160444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:45.160460 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:45.160540 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:45.160545 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:45.161897 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:45.161900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:45.162687 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:45.162696 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:45.163967 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:45.163971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:45.164079 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:45.164102 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 14:57:45.164175 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 14:57:45.164245 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":115,"threads_started":1}
I20260927 14:57:45.164588 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:45.159007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36157 (local address 127.0.0.1:34738)
0927 14:57:45.159556 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:45.159569 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:45.159645 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:45.160202 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:45.160206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:45.160222 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:45.160321 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:45.160328 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:45.161618 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:45.161622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:45.162880 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:45.162891 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:45.164201 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:45.164204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:45.164290 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:45.164323 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 14:57:45.164396 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 14:57:45.164464 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":4,"thread_start_us":85,"threads_started":1}
I20260927 14:57:45.164665 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:45.159538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34738 (local address 127.0.98.3:36157)
0927 14:57:45.159835 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:45.159846 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:45.159868 (+ 22us) server_negotiation.cc:407] Connection header received
0927 14:57:45.159927 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:45.159931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:45.160075 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:45.160203 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:45.160411 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:45.161462 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:45.163007 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:45.163768 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:45.164209 (+ 441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:45.164346 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:45.164430 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 14:57:45.164532 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":125,"threads_started":1}
I20260927 14:57:45.165006 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:45.159397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53586 (local address 127.0.98.1:35781)
0927 14:57:45.159882 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:45.159897 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:45.159918 (+ 21us) server_negotiation.cc:407] Connection header received
0927 14:57:45.159988 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:45.159992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:45.160164 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:45.160295 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:45.160655 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:45.161759 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:45.162840 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:45.163661 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:45.163766 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:45.163906 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:45.164750 (+ 844us) server_negotiation.cc:299] Negotiation successful
0927 14:57:45.164859 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":88,"threads_started":1}
I20260927 14:57:45.167374 10881 tablet_service.cc:1467] Processing CreateTablet for tablet 86384cf45ff24594b1414d47b32b91be (DEFAULT_TABLE table=test-table [id=d350c3936a824ab3a423ca6921026641]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:57:45.167709 11015 tablet_service.cc:1467] Processing CreateTablet for tablet 86384cf45ff24594b1414d47b32b91be (DEFAULT_TABLE table=test-table [id=d350c3936a824ab3a423ca6921026641]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:57:45.167881 10747 tablet_service.cc:1467] Processing CreateTablet for tablet 86384cf45ff24594b1414d47b32b91be (DEFAULT_TABLE table=test-table [id=d350c3936a824ab3a423ca6921026641]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:57:45.168632 11015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86384cf45ff24594b1414d47b32b91be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:45.168628 10881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86384cf45ff24594b1414d47b32b91be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:45.168776 10747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86384cf45ff24594b1414d47b32b91be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:57:45.175704 11098 tablet_bootstrap.cc:492] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09: Bootstrap starting.
I20260927 14:57:45.176360 11099 tablet_bootstrap.cc:492] T 86384cf45ff24594b1414d47b32b91be P d8be40d5917041069d71a3419bd68958: Bootstrap starting.
I20260927 14:57:45.177284 11100 tablet_bootstrap.cc:492] T 86384cf45ff24594b1414d47b32b91be P 080b23a41a12406aafece26788b802ff: Bootstrap starting.
I20260927 14:57:45.178457 11098 tablet_bootstrap.cc:654] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:45.178921 11099 tablet_bootstrap.cc:654] T 86384cf45ff24594b1414d47b32b91be P d8be40d5917041069d71a3419bd68958: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:45.179301 11100 tablet_bootstrap.cc:654] T 86384cf45ff24594b1414d47b32b91be P 080b23a41a12406aafece26788b802ff: Neither blocks nor log segments found. Creating new log.
I20260927 14:57:45.179831 11098 log.cc:824] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09: Log is configured to *not* fsync() on all Append() calls
I20260927 14:57:45.179826 11099 log.cc:824] T 86384cf45ff24594b1414d47b32b91be P d8be40d5917041069d71a3419bd68958: Log is configured to *not* fsync() on all Append() calls
I20260927 14:57:45.180478 11100 log.cc:824] T 86384cf45ff24594b1414d47b32b91be P 080b23a41a12406aafece26788b802ff: Log is configured to *not* fsync() on all Append() calls
I20260927 14:57:45.182035 11098 tablet_bootstrap.cc:492] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09: No bootstrap required, opened a new log
I20260927 14:57:45.182044 11099 tablet_bootstrap.cc:492] T 86384cf45ff24594b1414d47b32b91be P d8be40d5917041069d71a3419bd68958: No bootstrap required, opened a new log
I20260927 14:57:45.182240 11098 ts_tablet_manager.cc:1397] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260927 14:57:45.182241 11099 ts_tablet_manager.cc:1397] T 86384cf45ff24594b1414d47b32b91be P d8be40d5917041069d71a3419bd68958: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 14:57:45.182307 11100 tablet_bootstrap.cc:492] T 86384cf45ff24594b1414d47b32b91be P 080b23a41a12406aafece26788b802ff: No bootstrap required, opened a new log
I20260927 14:57:45.182475 11100 ts_tablet_manager.cc:1397] T 86384cf45ff24594b1414d47b32b91be P 080b23a41a12406aafece26788b802ff: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 14:57:45.185559 11098 raft_consensus.cc:348] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8be40d5917041069d71a3419bd68958" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "080b23a41a12406aafece26788b802ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35781 } } peers { permanent_uuid: "2a2ec4938faa483ca684479c870fcb09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36157 } }
I20260927 14:57:45.185945 11098 raft_consensus.cc:374] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:45.185999 11098 raft_consensus.cc:729] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a2ec4938faa483ca684479c870fcb09, State: Initialized, Role: FOLLOWER
I20260927 14:57:45.186276 11099 raft_consensus.cc:348] T 86384cf45ff24594b1414d47b32b91be P d8be40d5917041069d71a3419bd68958 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8be40d5917041069d71a3419bd68958" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "080b23a41a12406aafece26788b802ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35781 } } peers { permanent_uuid: "2a2ec4938faa483ca684479c870fcb09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36157 } }
I20260927 14:57:45.186276 11100 raft_consensus.cc:348] T 86384cf45ff24594b1414d47b32b91be P 080b23a41a12406aafece26788b802ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8be40d5917041069d71a3419bd68958" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "080b23a41a12406aafece26788b802ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35781 } } peers { permanent_uuid: "2a2ec4938faa483ca684479c870fcb09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36157 } }
I20260927 14:57:45.186563 11099 raft_consensus.cc:374] T 86384cf45ff24594b1414d47b32b91be P d8be40d5917041069d71a3419bd68958 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:45.186564 11100 raft_consensus.cc:374] T 86384cf45ff24594b1414d47b32b91be P 080b23a41a12406aafece26788b802ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:57:45.186493 11098 consensus_queue.cc:260] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8be40d5917041069d71a3419bd68958" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "080b23a41a12406aafece26788b802ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35781 } } peers { permanent_uuid: "2a2ec4938faa483ca684479c870fcb09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36157 } }
I20260927 14:57:45.186614 11099 raft_consensus.cc:729] T 86384cf45ff24594b1414d47b32b91be P d8be40d5917041069d71a3419bd68958 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8be40d5917041069d71a3419bd68958, State: Initialized, Role: FOLLOWER
I20260927 14:57:45.186614 11100 raft_consensus.cc:729] T 86384cf45ff24594b1414d47b32b91be P 080b23a41a12406aafece26788b802ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 080b23a41a12406aafece26788b802ff, State: Initialized, Role: FOLLOWER
I20260927 14:57:45.187151 11100 consensus_queue.cc:260] T 86384cf45ff24594b1414d47b32b91be P 080b23a41a12406aafece26788b802ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8be40d5917041069d71a3419bd68958" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "080b23a41a12406aafece26788b802ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35781 } } peers { permanent_uuid: "2a2ec4938faa483ca684479c870fcb09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36157 } }
I20260927 14:57:45.187151 11099 consensus_queue.cc:260] T 86384cf45ff24594b1414d47b32b91be P d8be40d5917041069d71a3419bd68958 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8be40d5917041069d71a3419bd68958" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "080b23a41a12406aafece26788b802ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35781 } } peers { permanent_uuid: "2a2ec4938faa483ca684479c870fcb09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36157 } }
I20260927 14:57:45.187515 11081 heartbeater.cc:503] Master 127.0.98.62:42451 was elected leader, sending a full tablet report...
I20260927 14:57:45.187683 11098 ts_tablet_manager.cc:1428] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
I20260927 14:57:45.188131 11100 ts_tablet_manager.cc:1428] T 86384cf45ff24594b1414d47b32b91be P 080b23a41a12406aafece26788b802ff: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 14:57:45.188131 11099 ts_tablet_manager.cc:1428] T 86384cf45ff24594b1414d47b32b91be P d8be40d5917041069d71a3419bd68958: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 14:57:45.188525 10947 heartbeater.cc:503] Master 127.0.98.62:42451 was elected leader, sending a full tablet report...
I20260927 14:57:45.188980 10813 heartbeater.cc:503] Master 127.0.98.62:42451 was elected leader, sending a full tablet report...
I20260927 14:57:45.330253 11104 raft_consensus.cc:482] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:57:45.330459 11104 raft_consensus.cc:504] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8be40d5917041069d71a3419bd68958" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "080b23a41a12406aafece26788b802ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35781 } } peers { permanent_uuid: "2a2ec4938faa483ca684479c870fcb09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36157 } }
I20260927 14:57:45.331604 11104 leader_election.cc:288] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8be40d5917041069d71a3419bd68958 (127.0.98.2:33481), 080b23a41a12406aafece26788b802ff (127.0.98.1:35781)
I20260927 14:57:45.335904 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:45.331926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33481 (local address 127.0.98.3:52107)
0927 14:57:45.332064 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:45.332083 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:45.332203 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:45.332531 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:45.332534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:45.332558 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:45.332696 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:45.332704 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:45.333658 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:45.333662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:45.334633 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:45.334643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:45.335466 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:45.335470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:45.335583 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:45.335611 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 14:57:45.335684 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 14:57:45.335768 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":21}
I20260927 14:57:45.335965 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:45.332062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52107 (local address 127.0.98.2:33481)
0927 14:57:45.332197 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:45.332202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:45.332218 (+ 16us) server_negotiation.cc:407] Connection header received
0927 14:57:45.332317 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:45.332320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:45.332366 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:45.332439 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:45.332837 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:45.333567 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:45.334768 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:45.335330 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:45.335415 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:45.335471 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:45.335750 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 14:57:45.335838 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 14:57:45.336352 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:45.332203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35781 (local address 127.0.98.3:52025)
0927 14:57:45.332518 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:45.332531 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:57:45.332621 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:57:45.332966 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:57:45.332969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:57:45.332988 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:57:45.333098 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:45.333104 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:45.334052 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:45.334055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:45.334964 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:57:45.334975 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:57:45.335760 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:57:45.335764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:57:45.335864 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:45.335889 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 14:57:45.336133 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 14:57:45.336213 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":1,"thread_start_us":130,"threads_started":1}
I20260927 14:57:45.336362 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:45.332327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52025 (local address 127.0.98.1:35781)
0927 14:57:45.332480 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:45.332484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:45.332605 (+ 121us) server_negotiation.cc:407] Connection header received
0927 14:57:45.332699 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:45.332701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:45.332851 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:45.332944 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:57:45.333279 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:45.333953 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:45.335121 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:45.335586 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:45.335714 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:45.335771 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:45.336166 (+ 395us) server_negotiation.cc:299] Negotiation successful
0927 14:57:45.336252 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 14:57:45.336851 10901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86384cf45ff24594b1414d47b32b91be" candidate_uuid: "2a2ec4938faa483ca684479c870fcb09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8be40d5917041069d71a3419bd68958" is_pre_election: true
I20260927 14:57:45.337006 10767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86384cf45ff24594b1414d47b32b91be" candidate_uuid: "2a2ec4938faa483ca684479c870fcb09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "080b23a41a12406aafece26788b802ff" is_pre_election: true
I20260927 14:57:45.337190 10901 raft_consensus.cc:2454] T 86384cf45ff24594b1414d47b32b91be P d8be40d5917041069d71a3419bd68958 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a2ec4938faa483ca684479c870fcb09 in term 0.
I20260927 14:57:45.337255 10767 raft_consensus.cc:2454] T 86384cf45ff24594b1414d47b32b91be P 080b23a41a12406aafece26788b802ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a2ec4938faa483ca684479c870fcb09 in term 0.
I20260927 14:57:45.337654 10971 leader_election.cc:302] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a2ec4938faa483ca684479c870fcb09, d8be40d5917041069d71a3419bd68958; no voters:
I20260927 14:57:45.337910 11104 raft_consensus.cc:2781] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:57:45.337989 11104 raft_consensus.cc:482] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:57:45.338018 11104 raft_consensus.cc:3037] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:45.339119 11104 raft_consensus.cc:504] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8be40d5917041069d71a3419bd68958" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "080b23a41a12406aafece26788b802ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35781 } } peers { permanent_uuid: "2a2ec4938faa483ca684479c870fcb09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36157 } }
I20260927 14:57:45.339485 11104 leader_election.cc:288] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09 [CANDIDATE]: Term 1 election: Requested vote from peers d8be40d5917041069d71a3419bd68958 (127.0.98.2:33481), 080b23a41a12406aafece26788b802ff (127.0.98.1:35781)
I20260927 14:57:45.339913 10901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86384cf45ff24594b1414d47b32b91be" candidate_uuid: "2a2ec4938faa483ca684479c870fcb09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8be40d5917041069d71a3419bd68958"
I20260927 14:57:45.339982 10767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86384cf45ff24594b1414d47b32b91be" candidate_uuid: "2a2ec4938faa483ca684479c870fcb09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "080b23a41a12406aafece26788b802ff"
I20260927 14:57:45.340031 10901 raft_consensus.cc:3037] T 86384cf45ff24594b1414d47b32b91be P d8be40d5917041069d71a3419bd68958 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:45.340130 10767 raft_consensus.cc:3037] T 86384cf45ff24594b1414d47b32b91be P 080b23a41a12406aafece26788b802ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:57:45.341136 10901 raft_consensus.cc:2454] T 86384cf45ff24594b1414d47b32b91be P d8be40d5917041069d71a3419bd68958 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a2ec4938faa483ca684479c870fcb09 in term 1.
I20260927 14:57:45.341208 10767 raft_consensus.cc:2454] T 86384cf45ff24594b1414d47b32b91be P 080b23a41a12406aafece26788b802ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a2ec4938faa483ca684479c870fcb09 in term 1.
I20260927 14:57:45.341452 10971 leader_election.cc:302] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a2ec4938faa483ca684479c870fcb09, d8be40d5917041069d71a3419bd68958; no voters:
I20260927 14:57:45.341650 11104 raft_consensus.cc:2781] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:57:45.341881 11104 raft_consensus.cc:686] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09 [term 1 LEADER]: Becoming Leader. State: Replica: 2a2ec4938faa483ca684479c870fcb09, State: Running, Role: LEADER
I20260927 14:57:45.342258 11104 consensus_queue.cc:237] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8be40d5917041069d71a3419bd68958" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } } peers { permanent_uuid: "080b23a41a12406aafece26788b802ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35781 } } peers { permanent_uuid: "2a2ec4938faa483ca684479c870fcb09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36157 } }
I20260927 14:57:45.345712 10621 catalog_manager.cc:5495] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a2ec4938faa483ca684479c870fcb09 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2a2ec4938faa483ca684479c870fcb09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8be40d5917041069d71a3419bd68958" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "080b23a41a12406aafece26788b802ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a2ec4938faa483ca684479c870fcb09" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36157 } health_report { overall_health: HEALTHY } } }
I20260927 14:57:45.360491 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:45.356875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34754 (local address 127.0.98.3:36157)
0927 14:57:45.357045 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 14:57:45.357048 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:57:45.357062 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:57:45.357136 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:57:45.357138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:57:45.357195 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:57:45.357299 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:57:45.357479 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:45.357971 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:45.358862 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:57:45.359091 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:57:45.359193 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:57:45.359598 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:57:45.360053 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:57:45.360163 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 14:57:45.360258 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 14:57:45.360317 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260927 14:57:45.363240 11082 tablet.cc:2369] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:57:45.366617 10767 raft_consensus.cc:1261] T 86384cf45ff24594b1414d47b32b91be P 080b23a41a12406aafece26788b802ff [term 1 FOLLOWER]: Refusing update from remote peer 2a2ec4938faa483ca684479c870fcb09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:57:45.366617 10901 raft_consensus.cc:1261] T 86384cf45ff24594b1414d47b32b91be P d8be40d5917041069d71a3419bd68958 [term 1 FOLLOWER]: Refusing update from remote peer 2a2ec4938faa483ca684479c870fcb09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:57:45.367472 11108 consensus_queue.cc:1035] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8be40d5917041069d71a3419bd68958" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:57:45.367671 11104 consensus_queue.cc:1035] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "080b23a41a12406aafece26788b802ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:57:45.376798 11113 mvcc.cc:205] Tried to move back new op lower bound from 7333974283733708800 to 7333974283644956672. Current Snapshot: MvccSnapshot[applied={T|T < 7333974283733708800}]
I20260927 14:57:45.377509 11112 mvcc.cc:205] Tried to move back new op lower bound from 7333974283733708800 to 7333974283644956672. Current Snapshot: MvccSnapshot[applied={T|T < 7333974283733708800}]
I20260927 14:57:45.384668 10620 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:57:45.384891 10620 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:57:45.387562 10620 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ab964e2dc66430ca699a1fd13706354: Sending DeleteTablet for 3 replicas of tablet 86384cf45ff24594b1414d47b32b91be
I20260927 14:57:45.388429 10881 tablet_service.cc:1514] Processing DeleteTablet for tablet 86384cf45ff24594b1414d47b32b91be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:57:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58834
I20260927 14:57:45.388482 10747 tablet_service.cc:1514] Processing DeleteTablet for tablet 86384cf45ff24594b1414d47b32b91be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:57:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53586
I20260927 14:57:45.389223 11121 tablet_replica.cc:330] T 86384cf45ff24594b1414d47b32b91be P 080b23a41a12406aafece26788b802ff: stopping tablet replica
I20260927 14:57:45.389183 11015 tablet_service.cc:1514] Processing DeleteTablet for tablet 86384cf45ff24594b1414d47b32b91be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:57:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34738
I20260927 14:57:45.389583 11121 raft_consensus.cc:2229] T 86384cf45ff24594b1414d47b32b91be P 080b23a41a12406aafece26788b802ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:57:45.389675 11123 tablet_replica.cc:330] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09: stopping tablet replica
I20260927 14:57:45.389940 11123 raft_consensus.cc:2229] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09 [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:57:45.390092 11121 raft_consensus.cc:2258] T 86384cf45ff24594b1414d47b32b91be P 080b23a41a12406aafece26788b802ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:57:45.390313 11123 raft_consensus.cc:2258] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:57:45.391829 11123 ts_tablet_manager.cc:1905] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:57:45.391829 11121 ts_tablet_manager.cc:1905] T 86384cf45ff24594b1414d47b32b91be P 080b23a41a12406aafece26788b802ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:57:45.392120 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 10684
I20260927 14:57:45.392141 11122 tablet_replica.cc:330] T 86384cf45ff24594b1414d47b32b91be P d8be40d5917041069d71a3419bd68958: stopping tablet replica
I20260927 14:57:45.392553 11122 raft_consensus.cc:2229] T 86384cf45ff24594b1414d47b32b91be P d8be40d5917041069d71a3419bd68958 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:57:45.393190 11122 raft_consensus.cc:2258] T 86384cf45ff24594b1414d47b32b91be P d8be40d5917041069d71a3419bd68958 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:57:45.394658 11122 ts_tablet_manager.cc:1905] T 86384cf45ff24594b1414d47b32b91be P d8be40d5917041069d71a3419bd68958: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:57:45.395224 11123 ts_tablet_manager.cc:1918] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:57:45.395328 11123 log.cc:1196] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-2/wal/wals/86384cf45ff24594b1414d47b32b91be
I20260927 14:57:45.395639 11123 ts_tablet_manager.cc:1939] T 86384cf45ff24594b1414d47b32b91be P 2a2ec4938faa483ca684479c870fcb09: Deleting consensus metadata
I20260927 14:57:45.397225 10609 catalog_manager.cc:4841] TS 2a2ec4938faa483ca684479c870fcb09 (127.0.98.3:36157): tablet 86384cf45ff24594b1414d47b32b91be (table test-table [id=d350c3936a824ab3a423ca6921026641]) successfully deleted
I20260927 14:57:45.397609 11122 ts_tablet_manager.cc:1918] T 86384cf45ff24594b1414d47b32b91be P d8be40d5917041069d71a3419bd68958: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:57:45.397812 11122 log.cc:1196] T 86384cf45ff24594b1414d47b32b91be P d8be40d5917041069d71a3419bd68958: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790521004237705-392-0/minicluster-data/ts-1/wal/wals/86384cf45ff24594b1414d47b32b91be
I20260927 14:57:45.398089 11122 ts_tablet_manager.cc:1939] T 86384cf45ff24594b1414d47b32b91be P d8be40d5917041069d71a3419bd68958: Deleting consensus metadata
I20260927 14:57:45.399266 10609 catalog_manager.cc:4841] TS d8be40d5917041069d71a3419bd68958 (127.0.98.2:33481): tablet 86384cf45ff24594b1414d47b32b91be (table test-table [id=d350c3936a824ab3a423ca6921026641]) successfully deleted
W20260927 14:57:45.407021 10609 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35781 (error 108)
I20260927 14:57:45.407215 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 10818
I20260927 14:57:45.408779 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:57:45.408420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35781 (local address 127.0.0.1:53598)
0927 14:57:45.408618 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 14:57:45.408689 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260927 14:57:45.408962 10609 catalog_manager.cc:4584] TS 080b23a41a12406aafece26788b802ff (127.0.98.1:35781): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 86384cf45ff24594b1414d47b32b91be: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35781: connect: Connection refused (error 111)
I20260927 14:57:45.415263 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 10952
I20260927 14:57:45.420516 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 10590
2026-09-27T14:57:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2287 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:57:45 dist-test-slave-pz0h krb5kdc[11127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:57:45 dist-test-slave-pz0h krb5kdc[11127](info): set up 2 sockets
Sep 27 14:57:45 dist-test-slave-pz0h krb5kdc[11127](info): commencing operation
krb5kdc: starting...
W20260927 14:57:46.380043 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 14:57:46 dist-test-slave-pz0h krb5kdc[11127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:57:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:57:46Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 14:57:46.545780 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskDguobN/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/postgres -l logfile start
2026-09-27 14:57:48.886 UTC [11157] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 14:57:48.886 UTC [11157] LOG: listening on IPv4 address "127.0.98.20", port 60409
2026-09-27 14:57:48.890 UTC [11157] LOG: listening on Unix socket "/tmp/.s.PGSQL.60409"
2026-09-27 14:57:48.918 UTC [11160] LOG: database system was shut down at 2026-09-27 14:57:47 UTC
2026-09-27 14:57:48.921 UTC [11157] LOG: database system is ready to accept connections
I20260927 14:57:49.787400 392 mini_postgres.cc:96] Postgres bound to 60409
2026-09-27 14:57:49.793 UTC [11168] FATAL: database "slave" does not exist
127.0.98.20:60409 - accepting connections
I20260927 14:57:49.794014 392 mini_ranger.cc:161] Starting Ranger...
I20260927 14:57:49.808947 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 14:57:50.297521 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 14:57:50.297621 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ranger-admin
2026-09-27 14:57:50,565 [I] DB FLAVOR :POSTGRES
2026-09-27 14:57:50,566 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 14:57:50,566 [I] Checking connection..
2026-09-27 14:57:50,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 14:57:50,826 [I] Checking connection passed.
2026-09-27 14:57:50,827 [I] --------- Verifying version history table ---------
2026-09-27 14:57:50,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 14:57:51,085 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 14:57:51,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 14:57:51,364 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 14:57:51,364 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 14:57:51,364 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60409/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 14:57:51.627 UTC [11232] WARNING: there is no transaction in progress
2026-09-27 14:57:51,633 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 14:57:51,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 14:57:51,909 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 14:57:51,909 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 14:57:51,909 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 14:57:52,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 14:57:52,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-pz0h.c.gcp-upstream.internal','N') ;"
2026-09-27 14:57:52,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 14:57:52,957 [I] Table x_portal_user does not exist in database ranger
2026-09-27 14:57:52,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 14:57:53,197 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 14:57:53,198 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 14:57:53,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 14:57:53,738 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 14:57:53,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60409/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 14:57:54.747 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 14:57:54.765 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 14:57:54.782 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 14:57:54.801 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 14:57:54.818 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 14:57:54.836 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 14:57:54.896 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 14:57:54.903 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 14:57:54.921 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 14:57:54.932 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 14:57:54.942 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 14:57:54.956 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 14:57:54.968 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 14:57:54.976 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 14:57:54.984 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 14:57:54.992 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 14:57:55,703 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 14:57:55,703 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 14:57:55,954 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 14:57:55,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 14:57:56,209 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 14:57:56,209 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 14:57:56,456 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 14:57:56,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 14:57:56,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 14:57:56,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-pz0h.c.gcp-upstream.internal';"
2026-09-27 14:57:57,209 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 14:57:57,210 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 14:57:57,447 [I] DB_PATCHES have already been applied
I20260927 14:57:57.454882 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskDguobN/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 14:57:57.454972 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 14:57:57.458963 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:46915
Sep 27, 2026 2:57:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:57:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:57:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 2:57:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 2:57:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 2:57:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 2:57:58 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 14:57:58.461508 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.668s user 0.000s sys 0.010s
Sep 27, 2026 2:58:19 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 2:58:19 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 2:58:19 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 2:58:19 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 2:58:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 2:58:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 2:58:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 2:58:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 2:58:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 2:58:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 2:58:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 2:58:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 2:58:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 2:58:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 2:58:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 2:58:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 2:58:19 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 2:58:20 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 14:58:21.390309 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 14:58:21 dist-test-slave-pz0h krb5kdc[11127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521101, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 14:58:21.476157 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 14:58:21.485836 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 14:58:22.287940 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 14:58:22.288070 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ranger-kms
I20260927 14:58:22.288167 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:60409
2026-09-27 14:58:22,477 [I] DB FLAVOR :POSTGRES
2026-09-27 14:58:22,477 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 14:58:22,478 [I] Checking connection
2026-09-27 14:58:22,478 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60409/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 14:58:22,722 [I] connection success
2026-09-27 14:58:22,722 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 14:58:22,722 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 14:58:22,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60409/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 14:58:22,972 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 14:58:22,972 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 14:58:22,972 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 14:58:22,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60409/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 14:58:23,259 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 14:58:23.829291 392 mini_ranger_kms.cc:310] Created kms service
I20260927 14:58:24.011838 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 14:58:24.054956 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 14:58:24 dist-test-slave-pz0h krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521101, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:58:24.524291 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 14:58:24.524438 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 14:58:24.524457 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 14:58:24.529233 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11521
I20260927 14:58:24.529395 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:36089
Sep 27, 2026 2:58:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:58:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:58:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 2:58:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 2:58:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 2:58:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 14:58:26.610404 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.134s user 0.012s sys 0.161s
I20260927 14:58:26.610512 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 14:58:26.610596 392 mini_ranger_kms.cc:401] 127.0.98.20:36089/kms/v1/keys
Sep 27 14:58:26 dist-test-slave-pz0h krb5kdc[11127](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790521106, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 14:58:27 dist-test-slave-pz0h krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521106, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 14:58:27 dist-test-slave-pz0h krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521106, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 14:58:30 dist-test-slave-pz0h krb5kdc[11127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:58:30.094341 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35965
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35965
--ranger_config_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36089/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:58:30.218163 11571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:58:30.218485 11571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:58:30.218544 11571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:58:30.222731 11571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:58:30.222849 11571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:58:30.222880 11571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:58:30.222908 11571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:58:30.222936 11571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:58:30.228905 11571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:36089/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35965
--ranger_config_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:58:30.230106 11571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:58:30.231225 11571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:58:30.240128 11577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:58:30.240114 11579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:58:30.240091 11576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:58:30.240360 11571 server_base.cc:1034] running on GCE node
I20260927 14:58:30.241207 11571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:58:30.242707 11571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:58:30.243899 11571 hybrid_clock.cc:648] HybridClock initialized: now 1790521110243871 us; error 45 us; skew 500 ppm
Sep 27 14:58:30 dist-test-slave-pz0h krb5kdc[11127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:58:30.247605 11571 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:58:30.248886 11571 webserver.cc:460] Webserver started at http://127.0.98.62:46319/ using document root <none> and password file <none>
I20260927 14:58:30.249646 11571 fs_manager.cc:359] Metadata directory not provided
I20260927 14:58:30.249737 11571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:58:30.249994 11571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 14:58:30 dist-test-slave-pz0h krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:58:30.299696 11571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "9b7ce625274f4b4ca3e109b33b7c8861"
format_stamp: "Formatted at 2026-09-27 14:58:30 on dist-test-slave-pz0h"
server_key: "5e12f2eb22c6a0f13a2a5bf01b9db591"
server_key_iv: "81eef59cc44f473cfe81b4ef3c5d7b34"
server_key_version: "kuduclusterkey@0"
I20260927 14:58:30.300498 11571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "9b7ce625274f4b4ca3e109b33b7c8861"
format_stamp: "Formatted at 2026-09-27 14:58:30 on dist-test-slave-pz0h"
server_key: "5e12f2eb22c6a0f13a2a5bf01b9db591"
server_key_iv: "81eef59cc44f473cfe81b4ef3c5d7b34"
server_key_version: "kuduclusterkey@0"
I20260927 14:58:30.305861 11571 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 14:58:30.323668 11586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:58:30.325381 11571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:58:30.325536 11571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "9b7ce625274f4b4ca3e109b33b7c8861"
format_stamp: "Formatted at 2026-09-27 14:58:30 on dist-test-slave-pz0h"
server_key: "5e12f2eb22c6a0f13a2a5bf01b9db591"
server_key_iv: "81eef59cc44f473cfe81b4ef3c5d7b34"
server_key_version: "kuduclusterkey@0"
I20260927 14:58:30.325680 11571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:58:30.368261 11571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:58:30.371739 11571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:58:30.371937 11571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:58:30.380455 11571 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35965
I20260927 14:58:30.380475 11638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35965 every 8 connection(s)
I20260927 14:58:30.381702 11571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:58:30.383006 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 11571
I20260927 14:58:30.383150 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 14:58:30.383473 392 external_mini_cluster.cc:1428] Setting key 7438d8c108ec8adb100071da31b79fbb
I20260927 14:58:30.386164 11639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:58:30.393307 11639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b7ce625274f4b4ca3e109b33b7c8861: Bootstrap starting.
Sep 27 14:58:30 dist-test-slave-pz0h krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:58:30.396045 11639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b7ce625274f4b4ca3e109b33b7c8861: Neither blocks nor log segments found. Creating new log.
I20260927 14:58:30.397250 11639 log.cc:824] T 00000000000000000000000000000000 P 9b7ce625274f4b4ca3e109b33b7c8861: Log is configured to *not* fsync() on all Append() calls
I20260927 14:58:30.399853 11639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b7ce625274f4b4ca3e109b33b7c8861: No bootstrap required, opened a new log
I20260927 14:58:30.400133 11642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:58:30.384935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43060 (local address 127.0.98.62:35965)
0927 14:58:30.385390 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0927 14:58:30.385409 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 14:58:30.385456 (+ 47us) server_negotiation.cc:407] Connection header received
0927 14:58:30.386324 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:58:30.386348 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:58:30.386784 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:58:30.387215 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:58:30.388110 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:58:30.389640 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:58:30.390849 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:58:30.391556 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:58:30.391718 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:58:30.394845 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:58:30.394853 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:58:30.394872 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:58:30.394912 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:58:30.396832 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:58:30.397371 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:58:30.397377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:58:30.397384 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:58:30.397517 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:58:30.397815 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:58:30.397819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:58:30.397821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:58:30.398299 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:58:30.398483 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 14:58:30.398836 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 14:58:30.399133 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":127,"threads_started":1}
I20260927 14:58:30.403427 11639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b7ce625274f4b4ca3e109b33b7c8861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7ce625274f4b4ca3e109b33b7c8861" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35965 } }
I20260927 14:58:30.403687 11639 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b7ce625274f4b4ca3e109b33b7c8861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:58:30.403723 11639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b7ce625274f4b4ca3e109b33b7c8861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b7ce625274f4b4ca3e109b33b7c8861, State: Initialized, Role: FOLLOWER
I20260927 14:58:30.404197 11639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b7ce625274f4b4ca3e109b33b7c8861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7ce625274f4b4ca3e109b33b7c8861" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35965 } }
I20260927 14:58:30.404376 11639 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b7ce625274f4b4ca3e109b33b7c8861 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:58:30.404456 11639 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b7ce625274f4b4ca3e109b33b7c8861 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:58:30.404559 11639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b7ce625274f4b4ca3e109b33b7c8861 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:58:30.405539 11639 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b7ce625274f4b4ca3e109b33b7c8861 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7ce625274f4b4ca3e109b33b7c8861" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35965 } }
I20260927 14:58:30.405889 11639 leader_election.cc:302] T 00000000000000000000000000000000 P 9b7ce625274f4b4ca3e109b33b7c8861 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b7ce625274f4b4ca3e109b33b7c8861; no voters:
I20260927 14:58:30.406198 11639 leader_election.cc:288] T 00000000000000000000000000000000 P 9b7ce625274f4b4ca3e109b33b7c8861 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:58:30.406316 11644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b7ce625274f4b4ca3e109b33b7c8861 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:58:30.406497 11644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b7ce625274f4b4ca3e109b33b7c8861 [term 1 LEADER]: Becoming Leader. State: Replica: 9b7ce625274f4b4ca3e109b33b7c8861, State: Running, Role: LEADER
I20260927 14:58:30.406908 11644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b7ce625274f4b4ca3e109b33b7c8861 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7ce625274f4b4ca3e109b33b7c8861" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35965 } }
I20260927 14:58:30.407624 11639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b7ce625274f4b4ca3e109b33b7c8861 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:58:30.408135 11644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b7ce625274f4b4ca3e109b33b7c8861 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b7ce625274f4b4ca3e109b33b7c8861. Latest consensus state: current_term: 1 leader_uuid: "9b7ce625274f4b4ca3e109b33b7c8861" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7ce625274f4b4ca3e109b33b7c8861" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35965 } } }
I20260927 14:58:30.408324 11644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b7ce625274f4b4ca3e109b33b7c8861 [sys.catalog]: This master's current role is: LEADER
I20260927 14:58:30.408506 11645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b7ce625274f4b4ca3e109b33b7c8861 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b7ce625274f4b4ca3e109b33b7c8861" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b7ce625274f4b4ca3e109b33b7c8861" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35965 } } }
I20260927 14:58:30.408675 11645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b7ce625274f4b4ca3e109b33b7c8861 [sys.catalog]: This master's current role is: LEADER
I20260927 14:58:30.408735 11650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:58:30.411994 11650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:58:30.414880 11639 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 14:58:30.422968 11650 catalog_manager.cc:1347] Generated new cluster ID: b4b76bd3d5c74cbe97245155a3116e27
I20260927 14:58:30.423087 11650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:58:30.440227 11650 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:58:30.442370 11650 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:58:30.450311 11650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b7ce625274f4b4ca3e109b33b7c8861: Generated new TSK 0
I20260927 14:58:30.452538 11650 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:58:31.165879 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36089/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35965
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:58:31.329030 11684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:58:31.329344 11684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:58:31.329389 11684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:58:31.334281 11684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:58:31.334357 11684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:58:31.334420 11684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:58:31.342026 11684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:36089/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:58:31.343291 11684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:58:31.344332 11684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:58:31.355599 11694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:58:31.355857 11692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:58:31.355599 11691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:58:31.356931 11684 server_base.cc:1034] running on GCE node
I20260927 14:58:31.357432 11684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:58:31.358229 11684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:58:31.359429 11684 hybrid_clock.cc:648] HybridClock initialized: now 1790521111359398 us; error 48 us; skew 500 ppm
Sep 27 14:58:31 dist-test-slave-pz0h krb5kdc[11127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:58:31.362221 11684 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:58:31.363273 11684 webserver.cc:460] Webserver started at http://127.0.98.1:40429/ using document root <none> and password file <none>
I20260927 14:58:31.363821 11684 fs_manager.cc:359] Metadata directory not provided
I20260927 14:58:31.363888 11684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:58:31.364082 11684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 14:58:31 dist-test-slave-pz0h krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:58:31.385061 11684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-0/data/instance:
uuid: "7d93bf0b8401424885bef354966cef6b"
format_stamp: "Formatted at 2026-09-27 14:58:31 on dist-test-slave-pz0h"
server_key: "55209d63a16f0ca979fad29537c9ebed"
server_key_iv: "8247324d1f80d43fcb29b1566948409d"
server_key_version: "kuduclusterkey@0"
I20260927 14:58:31.387640 11684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7d93bf0b8401424885bef354966cef6b"
format_stamp: "Formatted at 2026-09-27 14:58:31 on dist-test-slave-pz0h"
server_key: "55209d63a16f0ca979fad29537c9ebed"
server_key_iv: "8247324d1f80d43fcb29b1566948409d"
server_key_version: "kuduclusterkey@0"
I20260927 14:58:31.396435 11684 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260927 14:58:31.432374 11701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:58:31.440164 11684 fs_manager.cc:725] Time spent opening block manager: real 0.027s user 0.002s sys 0.000s
I20260927 14:58:31.440397 11684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "7d93bf0b8401424885bef354966cef6b"
format_stamp: "Formatted at 2026-09-27 14:58:31 on dist-test-slave-pz0h"
server_key: "55209d63a16f0ca979fad29537c9ebed"
server_key_iv: "8247324d1f80d43fcb29b1566948409d"
server_key_version: "kuduclusterkey@0"
I20260927 14:58:31.440568 11684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:58:31.468729 11684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:58:31.472605 11684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:58:31.472870 11684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:58:31.473685 11684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:58:31.474958 11684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:58:31.475063 11684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:58:31.475212 11684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:58:31.475308 11684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:58:31.487396 11684 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45101
I20260927 14:58:31.488581 11684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:58:31.489107 11816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45101 every 8 connection(s)
I20260927 14:58:31.495371 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 11684
I20260927 14:58:31.495501 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:58:31.495824 392 external_mini_cluster.cc:1428] Setting key 7f0ab7498b45268353d0f8bf1de3c1c7
Sep 27 14:58:31 dist-test-slave-pz0h krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:58:31.503670 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:58:31.490544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39073 (local address 127.0.98.62:35965)
0927 14:58:31.490857 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0927 14:58:31.490861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:58:31.491677 (+ 816us) server_negotiation.cc:407] Connection header received
0927 14:58:31.492691 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:58:31.492694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:58:31.492764 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:58:31.492843 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:58:31.493958 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:58:31.494506 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:58:31.496051 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:58:31.496467 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:58:31.496539 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:58:31.499617 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:58:31.499621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:58:31.499624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:58:31.499654 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:58:31.501303 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:58:31.501996 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:58:31.501999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:58:31.502000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:58:31.502081 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:58:31.502431 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:58:31.502434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:58:31.502435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:58:31.502623 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:58:31.502706 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 14:58:31.503347 (+ 641us) server_negotiation.cc:299] Negotiation successful
0927 14:58:31.503484 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":92,"threads_started":1}
I20260927 14:58:31.504678 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:58:31.491043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35965 (local address 127.0.98.1:39073)
0927 14:58:31.491520 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 14:58:31.491574 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 14:58:31.492476 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:58:31.493031 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:58:31.493038 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:58:31.493319 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:58:31.493802 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:58:31.493816 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:58:31.494632 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:58:31.494635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:58:31.495896 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:58:31.495908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:58:31.496752 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:58:31.496756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:58:31.496961 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:58:31.497658 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:58:31.497680 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:58:31.499392 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:58:31.501433 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:58:31.501439 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:58:31.501455 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:58:31.501892 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:58:31.502168 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:58:31.502171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:58:31.502174 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:58:31.502330 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:58:31.502731 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:58:31.502738 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:58:31.503094 (+ 356us) client_negotiation.cc:769] Sending connection context
0927 14:58:31.503275 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 14:58:31.503621 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":119,"threads_started":1}
I20260927 14:58:31.505944 11817 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35965
I20260927 14:58:31.506191 11817 heartbeater.cc:464] Registering TS with master...
I20260927 14:58:31.506711 11817 heartbeater.cc:511] Master 127.0.98.62:35965 requested a full tablet report, sending...
I20260927 14:58:31.508322 11603 ts_manager.cc:194] Registered new tserver with Master: 7d93bf0b8401424885bef354966cef6b (127.0.98.1:45101)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 14:58:31.509562 11603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:58:31.534082 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36089/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35965
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 14:58:31 dist-test-slave-pz0h krb5kdc[11127](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790521111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 14:58:31.684103 11823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:58:31.684418 11823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:58:31.684481 11823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:58:31.688967 11823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:58:31.689045 11823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:58:31.689128 11823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:58:31.696013 11823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:36089/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:58:31.697139 11823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:58:31.698323 11823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:58:31.708995 11829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:58:31.708977 11830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:58:31.708981 11832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:58:31.709489 11823 server_base.cc:1034] running on GCE node
I20260927 14:58:31.710076 11823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:58:31.710759 11823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:58:31.711963 11823 hybrid_clock.cc:648] HybridClock initialized: now 1790521111711928 us; error 45 us; skew 500 ppm
Sep 27 14:58:31 dist-test-slave-pz0h krb5kdc[11127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:58:31.715427 11823 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:58:31.716625 11823 webserver.cc:460] Webserver started at http://127.0.98.2:35731/ using document root <none> and password file <none>
I20260927 14:58:31.717228 11823 fs_manager.cc:359] Metadata directory not provided
I20260927 14:58:31.717303 11823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:58:31.717535 11823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 14:58:31 dist-test-slave-pz0h krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:58:31.736040 11823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-1/data/instance:
uuid: "d01b4849a8244207a214ad55e53895bd"
format_stamp: "Formatted at 2026-09-27 14:58:31 on dist-test-slave-pz0h"
server_key: "da18d6c7fd9726c5b3cdc45a2c77607c"
server_key_iv: "29d8e0597128c78dbfd96136d744e96d"
server_key_version: "kuduclusterkey@0"
I20260927 14:58:31.736701 11823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d01b4849a8244207a214ad55e53895bd"
format_stamp: "Formatted at 2026-09-27 14:58:31 on dist-test-slave-pz0h"
server_key: "da18d6c7fd9726c5b3cdc45a2c77607c"
server_key_iv: "29d8e0597128c78dbfd96136d744e96d"
server_key_version: "kuduclusterkey@0"
I20260927 14:58:31.741056 11823 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 14:58:31.766315 11839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:58:31.768175 11823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:58:31.768356 11823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "d01b4849a8244207a214ad55e53895bd"
format_stamp: "Formatted at 2026-09-27 14:58:31 on dist-test-slave-pz0h"
server_key: "da18d6c7fd9726c5b3cdc45a2c77607c"
server_key_iv: "29d8e0597128c78dbfd96136d744e96d"
server_key_version: "kuduclusterkey@0"
I20260927 14:58:31.768513 11823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:58:31.803651 11823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:58:31.807677 11823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:58:31.807997 11823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:58:31.808950 11823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:58:31.810246 11823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:58:31.810371 11823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:58:31.810472 11823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:58:31.810534 11823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:58:31.827469 11823 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41907
I20260927 14:58:31.828440 11823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:58:31.830739 11952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41907 every 8 connection(s)
I20260927 14:58:31.835760 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 11823
I20260927 14:58:31.835907 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:58:31.836230 392 external_mini_cluster.cc:1428] Setting key f032fcedd7bd0cef99e7ee70065d4a56
Sep 27 14:58:31 dist-test-slave-pz0h krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:58:31.847424 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:58:31.832588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40711 (local address 127.0.98.62:35965)
0927 14:58:31.832739 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 14:58:31.832743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:58:31.833962 (+ 1219us) server_negotiation.cc:407] Connection header received
0927 14:58:31.835255 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:58:31.835260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:58:31.835319 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:58:31.835419 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:58:31.836864 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:58:31.837610 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:58:31.839287 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:58:31.839688 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:58:31.839805 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:58:31.843084 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:58:31.843088 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:58:31.843090 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:58:31.843123 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:58:31.844800 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:58:31.845505 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:58:31.845509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:58:31.845511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:58:31.845606 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:58:31.846022 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:58:31.846025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:58:31.846027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:58:31.846254 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:58:31.846329 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 14:58:31.847099 (+ 770us) server_negotiation.cc:299] Negotiation successful
0927 14:58:31.847249 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 14:58:31.849422 11955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:58:31.833139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35965 (local address 127.0.98.2:40711)
0927 14:58:31.833788 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0927 14:58:31.833848 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 14:58:31.834971 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:58:31.835622 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:58:31.835631 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:58:31.835981 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:58:31.836680 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:58:31.836703 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:58:31.837773 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:58:31.837777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:58:31.839139 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:58:31.839152 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:58:31.840012 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:58:31.840016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:58:31.840264 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:58:31.841009 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:58:31.841033 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:58:31.842834 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:58:31.844973 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:58:31.844980 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:58:31.845002 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:58:31.845378 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:58:31.845709 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:58:31.845711 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:58:31.845713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:58:31.845917 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:58:31.846395 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:58:31.846403 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:58:31.846789 (+ 386us) client_negotiation.cc:769] Sending connection context
0927 14:58:31.847924 (+ 1135us) client_negotiation.cc:240] Negotiation successful
0927 14:58:31.848316 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":180,"threads_started":1}
I20260927 14:58:31.850939 11953 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 14:58:31.851277 11953 heartbeater.cc:464] Registering TS with master...
I20260927 14:58:31.851970 11953 heartbeater.cc:511] Master 127.0.98.62:35965 requested a full tablet report, sending...
I20260927 14:58:31.853356 11603 ts_manager.cc:194] Registered new tserver with Master: d01b4849a8244207a214ad55e53895bd (127.0.98.2:41907)
I20260927 14:58:31.855919 11603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:58:31.886454 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36089/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35965
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:58:32.060663 11958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:58:32.061200 11958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:58:32.061328 11958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:58:32.067799 11958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:58:32.067970 11958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:58:32.068094 11958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:58:32.076328 11958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:36089/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:58:32.078008 11958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:58:32.079522 11958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:58:32.093415 11966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:58:32.093885 11958 server_base.cc:1034] running on GCE node
W20260927 14:58:32.094182 11964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:58:32.094408 11963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:58:32.095003 11958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:58:32.095881 11958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:58:32.097043 11958 hybrid_clock.cc:648] HybridClock initialized: now 1790521112096981 us; error 95 us; skew 500 ppm
Sep 27 14:58:32 dist-test-slave-pz0h krb5kdc[11127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:58:32.100826 11958 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:58:32.102031 11958 webserver.cc:460] Webserver started at http://127.0.98.3:42311/ using document root <none> and password file <none>
I20260927 14:58:32.102888 11958 fs_manager.cc:359] Metadata directory not provided
I20260927 14:58:32.103082 11958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:58:32.103416 11958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 14:58:32 dist-test-slave-pz0h krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 14:58:32 dist-test-slave-pz0h krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:58:32.123760 11958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-2/data/instance:
uuid: "a048bdb8c3254c83b32dd5cab052c41e"
format_stamp: "Formatted at 2026-09-27 14:58:32 on dist-test-slave-pz0h"
server_key: "643dd40ec0e2bbb41284eb41d539e26c"
server_key_iv: "08a22c9b67bf158e330289ba3865250e"
server_key_version: "kuduclusterkey@0"
I20260927 14:58:32.125674 11958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a048bdb8c3254c83b32dd5cab052c41e"
format_stamp: "Formatted at 2026-09-27 14:58:32 on dist-test-slave-pz0h"
server_key: "643dd40ec0e2bbb41284eb41d539e26c"
server_key_iv: "08a22c9b67bf158e330289ba3865250e"
server_key_version: "kuduclusterkey@0"
I20260927 14:58:32.130882 11958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 14:58:32.165301 11973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:58:32.167289 11958 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 14:58:32.167439 11958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "a048bdb8c3254c83b32dd5cab052c41e"
format_stamp: "Formatted at 2026-09-27 14:58:32 on dist-test-slave-pz0h"
server_key: "643dd40ec0e2bbb41284eb41d539e26c"
server_key_iv: "08a22c9b67bf158e330289ba3865250e"
server_key_version: "kuduclusterkey@0"
I20260927 14:58:32.167572 11958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:58:32.194504 11958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:58:32.198096 11958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:58:32.198359 11958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:58:32.199090 11958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:58:32.200184 11958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:58:32.200300 11958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:58:32.200392 11958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:58:32.200453 11958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:58:32.213774 11958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36747
I20260927 14:58:32.216526 11958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:58:32.218696 12086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36747 every 8 connection(s)
I20260927 14:58:32.222564 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 11958
I20260927 14:58:32.222702 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:58:32.223516 392 external_mini_cluster.cc:1428] Setting key 4e17fe24eac8919e38aec16bff13c846
Sep 27 14:58:32 dist-test-slave-pz0h krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:58:32.242344 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:58:32.222934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59383 (local address 127.0.98.62:35965)
0927 14:58:32.223221 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 14:58:32.223226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:58:32.226657 (+ 3431us) server_negotiation.cc:407] Connection header received
0927 14:58:32.227966 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:58:32.227970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:58:32.228027 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:58:32.228077 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:58:32.229803 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:58:32.230621 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:58:32.232433 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:58:32.232769 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:58:32.232845 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:58:32.237654 (+ 4809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:58:32.237658 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:58:32.237661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:58:32.237692 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:58:32.239489 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:58:32.240419 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:58:32.240424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:58:32.240425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:58:32.240519 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:58:32.240926 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:58:32.240929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:58:32.240931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:58:32.241161 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:58:32.241250 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 14:58:32.242024 (+ 774us) server_negotiation.cc:299] Negotiation successful
0927 14:58:32.242163 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260927 14:58:32.244411 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:58:32.221170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35965 (local address 127.0.98.3:59383)
0927 14:58:32.226494 (+ 5324us) negotiation.cc:107] Waiting for socket to connect
0927 14:58:32.226561 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 14:58:32.227702 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:58:32.228544 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:58:32.228555 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:58:32.228940 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:58:32.229631 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:58:32.229649 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:58:32.230830 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:58:32.230834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:58:32.232288 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:58:32.232302 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:58:32.233063 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:58:32.233066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:58:32.233279 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:58:32.233989 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:58:32.234011 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:58:32.237412 (+ 3401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:58:32.239833 (+ 2421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:58:32.239840 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:58:32.239861 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:58:32.240257 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:58:32.240625 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:58:32.240628 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:58:32.240631 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:58:32.240826 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:58:32.241318 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:58:32.241325 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:58:32.241740 (+ 415us) client_negotiation.cc:769] Sending connection context
0927 14:58:32.242919 (+ 1179us) client_negotiation.cc:240] Negotiation successful
0927 14:58:32.243339 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5073,"thread_start_us":177,"threads_started":1}
I20260927 14:58:32.245980 12087 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35965
I20260927 14:58:32.246237 12087 heartbeater.cc:464] Registering TS with master...
I20260927 14:58:32.246842 12087 heartbeater.cc:511] Master 127.0.98.62:35965 requested a full tablet report, sending...
I20260927 14:58:32.249532 11603 ts_manager.cc:194] Registered new tserver with Master: a048bdb8c3254c83b32dd5cab052c41e (127.0.98.3:36747)
I20260927 14:58:32.250286 11603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59383
I20260927 14:58:32.261286 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:58:32.274269 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:58:32.263845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43066 (local address 127.0.98.62:35965)
0927 14:58:32.263972 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 14:58:32.263976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:58:32.264627 (+ 651us) server_negotiation.cc:407] Connection header received
0927 14:58:32.264816 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:58:32.264820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:58:32.264874 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:58:32.264926 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:58:32.266145 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:58:32.266960 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:58:32.268127 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:58:32.268450 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:58:32.268528 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:58:32.270347 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:58:32.270351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:58:32.270354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:58:32.270387 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:58:32.272046 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:58:32.272568 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:58:32.272572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:58:32.272573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:58:32.272667 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:58:32.272967 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:58:32.272971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:58:32.272972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:58:32.273265 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:58:32.273379 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 14:58:32.273941 (+ 562us) server_negotiation.cc:299] Negotiation successful
0927 14:58:32.274092 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:58:32.278056 11603 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:58:32.282709 11603 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:58:32.308732 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:58:32.299349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36747 (local address 127.0.0.1:33066)
0927 14:58:32.301815 (+ 2466us) negotiation.cc:107] Waiting for socket to connect
0927 14:58:32.301828 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:58:32.301920 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:58:32.302654 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:58:32.302661 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:58:32.302685 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:58:32.302871 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:58:32.302878 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:58:32.305108 (+ 2230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:58:32.305112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:58:32.306654 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:58:32.306667 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:58:32.308052 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:58:32.308057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:58:32.308196 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:58:32.308257 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 14:58:32.308377 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 14:58:32.308538 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2406,"thread_start_us":85,"threads_started":1}
I20260927 14:58:32.309437 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:58:32.299732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33066 (local address 127.0.98.3:36747)
0927 14:58:32.300954 (+ 1222us) server_negotiation.cc:206] Beginning negotiation
0927 14:58:32.300967 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 14:58:32.301914 (+ 947us) server_negotiation.cc:407] Connection header received
0927 14:58:32.302040 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:58:32.302046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:58:32.302304 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:58:32.302472 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:58:32.303000 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:58:32.304977 (+ 1977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:58:32.306856 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:58:32.307772 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:58:32.307892 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:58:32.309052 (+ 1160us) server_negotiation.cc:1035] Waiting for connection context
0927 14:58:32.309117 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 14:58:32.309240 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1088,"thread_start_us":123,"threads_started":1}
I20260927 14:58:32.313529 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:58:32.297360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45101 (local address 127.0.0.1:41946)
0927 14:58:32.300433 (+ 3073us) negotiation.cc:107] Waiting for socket to connect
0927 14:58:32.300496 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 14:58:32.300632 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:58:32.302888 (+ 2256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:58:32.302892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:58:32.302911 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:58:32.303004 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:58:32.303010 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:58:32.305667 (+ 2657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:58:32.305670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:58:32.306911 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:58:32.306922 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:58:32.313027 (+ 6105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:58:32.313032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:58:32.313162 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:58:32.313194 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 14:58:32.313260 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 14:58:32.313355 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2882,"thread_start_us":113,"threads_started":1}
I20260927 14:58:32.313687 12021 tablet_service.cc:1467] Processing CreateTablet for tablet 82139f305e4147b88698289688314a01 (DEFAULT_TABLE table=test-table [id=c4634a1c569041a0a6092a6d1ec3a283]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:58:32.314560 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:58:32.298326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41946 (local address 127.0.98.1:45101)
0927 14:58:32.302265 (+ 3939us) server_negotiation.cc:206] Beginning negotiation
0927 14:58:32.302272 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:58:32.302298 (+ 26us) server_negotiation.cc:407] Connection header received
0927 14:58:32.302373 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:58:32.302379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:58:32.302572 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:58:32.302715 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:58:32.303156 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:58:32.305530 (+ 2374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:58:32.307080 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:58:32.311766 (+ 4686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:58:32.313967 (+ 2201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:58:32.314150 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 14:58:32.314223 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 14:58:32.314409 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3808,"thread_start_us":97,"threads_started":1}
I20260927 14:58:32.319948 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:58:32.297437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41907 (local address 127.0.0.1:43602)
0927 14:58:32.301269 (+ 3832us) negotiation.cc:107] Waiting for socket to connect
0927 14:58:32.301284 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:58:32.301619 (+ 335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:58:32.307418 (+ 5799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:58:32.307421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:58:32.307451 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:58:32.307553 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:58:32.307559 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:58:32.314876 (+ 7317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:58:32.314880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:58:32.317075 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:58:32.317090 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:58:32.319485 (+ 2395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:58:32.319489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:58:32.319606 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:58:32.319639 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 14:58:32.319694 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 14:58:32.319785 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3598,"thread_start_us":102,"threads_started":1}
I20260927 14:58:32.322045 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:58:32.298663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43602 (local address 127.0.98.2:41907)
0927 14:58:32.304503 (+ 5840us) server_negotiation.cc:206] Beginning negotiation
0927 14:58:32.304525 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 14:58:32.304552 (+ 27us) server_negotiation.cc:407] Connection header received
0927 14:58:32.304631 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:58:32.304637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:58:32.304868 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:58:32.307729 (+ 2861us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:58:32.307882 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:58:32.312838 (+ 4956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:58:32.317307 (+ 4469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:58:32.318250 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:58:32.320502 (+ 2252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:58:32.321690 (+ 1188us) server_negotiation.cc:1035] Waiting for connection context
0927 14:58:32.321765 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 14:58:32.321895 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5718,"thread_start_us":135,"threads_started":1}
I20260927 14:58:32.322109 12021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82139f305e4147b88698289688314a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:58:32.321877 11742 tablet_service.cc:1467] Processing CreateTablet for tablet 82139f305e4147b88698289688314a01 (DEFAULT_TABLE table=test-table [id=c4634a1c569041a0a6092a6d1ec3a283]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:58:32.323367 11742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82139f305e4147b88698289688314a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:58:32.325603 11872 tablet_service.cc:1467] Processing CreateTablet for tablet 82139f305e4147b88698289688314a01 (DEFAULT_TABLE table=test-table [id=c4634a1c569041a0a6092a6d1ec3a283]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:58:32.326848 11872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82139f305e4147b88698289688314a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:58:32.336985 12107 tablet_bootstrap.cc:492] T 82139f305e4147b88698289688314a01 P 7d93bf0b8401424885bef354966cef6b: Bootstrap starting.
I20260927 14:58:32.339754 12107 tablet_bootstrap.cc:654] T 82139f305e4147b88698289688314a01 P 7d93bf0b8401424885bef354966cef6b: Neither blocks nor log segments found. Creating new log.
I20260927 14:58:32.340233 12106 tablet_bootstrap.cc:492] T 82139f305e4147b88698289688314a01 P a048bdb8c3254c83b32dd5cab052c41e: Bootstrap starting.
I20260927 14:58:32.340214 12108 tablet_bootstrap.cc:492] T 82139f305e4147b88698289688314a01 P d01b4849a8244207a214ad55e53895bd: Bootstrap starting.
I20260927 14:58:32.342908 12108 tablet_bootstrap.cc:654] T 82139f305e4147b88698289688314a01 P d01b4849a8244207a214ad55e53895bd: Neither blocks nor log segments found. Creating new log.
I20260927 14:58:32.343859 12107 log.cc:824] T 82139f305e4147b88698289688314a01 P 7d93bf0b8401424885bef354966cef6b: Log is configured to *not* fsync() on all Append() calls
I20260927 14:58:32.344255 12108 log.cc:824] T 82139f305e4147b88698289688314a01 P d01b4849a8244207a214ad55e53895bd: Log is configured to *not* fsync() on all Append() calls
I20260927 14:58:32.344538 12106 tablet_bootstrap.cc:654] T 82139f305e4147b88698289688314a01 P a048bdb8c3254c83b32dd5cab052c41e: Neither blocks nor log segments found. Creating new log.
I20260927 14:58:32.345850 12106 log.cc:824] T 82139f305e4147b88698289688314a01 P a048bdb8c3254c83b32dd5cab052c41e: Log is configured to *not* fsync() on all Append() calls
I20260927 14:58:32.346740 12108 tablet_bootstrap.cc:492] T 82139f305e4147b88698289688314a01 P d01b4849a8244207a214ad55e53895bd: No bootstrap required, opened a new log
I20260927 14:58:32.347005 12108 ts_tablet_manager.cc:1397] T 82139f305e4147b88698289688314a01 P d01b4849a8244207a214ad55e53895bd: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260927 14:58:32.348307 12107 tablet_bootstrap.cc:492] T 82139f305e4147b88698289688314a01 P 7d93bf0b8401424885bef354966cef6b: No bootstrap required, opened a new log
I20260927 14:58:32.348506 12107 ts_tablet_manager.cc:1397] T 82139f305e4147b88698289688314a01 P 7d93bf0b8401424885bef354966cef6b: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260927 14:58:32.352650 12108 raft_consensus.cc:348] T 82139f305e4147b88698289688314a01 P d01b4849a8244207a214ad55e53895bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d93bf0b8401424885bef354966cef6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45101 } } peers { permanent_uuid: "d01b4849a8244207a214ad55e53895bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41907 } } peers { permanent_uuid: "a048bdb8c3254c83b32dd5cab052c41e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 } }
I20260927 14:58:32.353183 12108 raft_consensus.cc:374] T 82139f305e4147b88698289688314a01 P d01b4849a8244207a214ad55e53895bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:58:32.353060 12107 raft_consensus.cc:348] T 82139f305e4147b88698289688314a01 P 7d93bf0b8401424885bef354966cef6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d93bf0b8401424885bef354966cef6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45101 } } peers { permanent_uuid: "d01b4849a8244207a214ad55e53895bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41907 } } peers { permanent_uuid: "a048bdb8c3254c83b32dd5cab052c41e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 } }
I20260927 14:58:32.353282 12108 raft_consensus.cc:729] T 82139f305e4147b88698289688314a01 P d01b4849a8244207a214ad55e53895bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d01b4849a8244207a214ad55e53895bd, State: Initialized, Role: FOLLOWER
I20260927 14:58:32.353374 12107 raft_consensus.cc:374] T 82139f305e4147b88698289688314a01 P 7d93bf0b8401424885bef354966cef6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:58:32.353478 12107 raft_consensus.cc:729] T 82139f305e4147b88698289688314a01 P 7d93bf0b8401424885bef354966cef6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d93bf0b8401424885bef354966cef6b, State: Initialized, Role: FOLLOWER
I20260927 14:58:32.353884 12108 consensus_queue.cc:260] T 82139f305e4147b88698289688314a01 P d01b4849a8244207a214ad55e53895bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d93bf0b8401424885bef354966cef6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45101 } } peers { permanent_uuid: "d01b4849a8244207a214ad55e53895bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41907 } } peers { permanent_uuid: "a048bdb8c3254c83b32dd5cab052c41e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 } }
I20260927 14:58:32.353945 12107 consensus_queue.cc:260] T 82139f305e4147b88698289688314a01 P 7d93bf0b8401424885bef354966cef6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d93bf0b8401424885bef354966cef6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45101 } } peers { permanent_uuid: "d01b4849a8244207a214ad55e53895bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41907 } } peers { permanent_uuid: "a048bdb8c3254c83b32dd5cab052c41e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 } }
I20260927 14:58:32.355086 12108 ts_tablet_manager.cc:1428] T 82139f305e4147b88698289688314a01 P d01b4849a8244207a214ad55e53895bd: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260927 14:58:32.355587 11953 heartbeater.cc:503] Master 127.0.98.62:35965 was elected leader, sending a full tablet report...
I20260927 14:58:32.355690 12107 ts_tablet_manager.cc:1428] T 82139f305e4147b88698289688314a01 P 7d93bf0b8401424885bef354966cef6b: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260927 14:58:32.356030 11817 heartbeater.cc:503] Master 127.0.98.62:35965 was elected leader, sending a full tablet report...
I20260927 14:58:32.357671 12106 tablet_bootstrap.cc:492] T 82139f305e4147b88698289688314a01 P a048bdb8c3254c83b32dd5cab052c41e: No bootstrap required, opened a new log
I20260927 14:58:32.357964 12106 ts_tablet_manager.cc:1397] T 82139f305e4147b88698289688314a01 P a048bdb8c3254c83b32dd5cab052c41e: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.006s
I20260927 14:58:32.362361 12106 raft_consensus.cc:348] T 82139f305e4147b88698289688314a01 P a048bdb8c3254c83b32dd5cab052c41e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d93bf0b8401424885bef354966cef6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45101 } } peers { permanent_uuid: "d01b4849a8244207a214ad55e53895bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41907 } } peers { permanent_uuid: "a048bdb8c3254c83b32dd5cab052c41e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 } }
I20260927 14:58:32.362886 12106 raft_consensus.cc:374] T 82139f305e4147b88698289688314a01 P a048bdb8c3254c83b32dd5cab052c41e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:58:32.363003 12106 raft_consensus.cc:729] T 82139f305e4147b88698289688314a01 P a048bdb8c3254c83b32dd5cab052c41e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a048bdb8c3254c83b32dd5cab052c41e, State: Initialized, Role: FOLLOWER
I20260927 14:58:32.363663 12106 consensus_queue.cc:260] T 82139f305e4147b88698289688314a01 P a048bdb8c3254c83b32dd5cab052c41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d93bf0b8401424885bef354966cef6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45101 } } peers { permanent_uuid: "d01b4849a8244207a214ad55e53895bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41907 } } peers { permanent_uuid: "a048bdb8c3254c83b32dd5cab052c41e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 } }
I20260927 14:58:32.364980 12106 ts_tablet_manager.cc:1428] T 82139f305e4147b88698289688314a01 P a048bdb8c3254c83b32dd5cab052c41e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 14:58:32.365473 12087 heartbeater.cc:503] Master 127.0.98.62:35965 was elected leader, sending a full tablet report...
I20260927 14:58:32.379549 12113 raft_consensus.cc:482] T 82139f305e4147b88698289688314a01 P d01b4849a8244207a214ad55e53895bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:58:32.379784 12113 raft_consensus.cc:504] T 82139f305e4147b88698289688314a01 P d01b4849a8244207a214ad55e53895bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d93bf0b8401424885bef354966cef6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45101 } } peers { permanent_uuid: "d01b4849a8244207a214ad55e53895bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41907 } } peers { permanent_uuid: "a048bdb8c3254c83b32dd5cab052c41e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 } }
I20260927 14:58:32.380982 12113 leader_election.cc:288] T 82139f305e4147b88698289688314a01 P d01b4849a8244207a214ad55e53895bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d93bf0b8401424885bef354966cef6b (127.0.98.1:45101), a048bdb8c3254c83b32dd5cab052c41e (127.0.98.3:36747)
I20260927 14:58:32.388594 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:58:32.381708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55455 (local address 127.0.98.1:45101)
0927 14:58:32.381831 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 14:58:32.381836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:58:32.382201 (+ 365us) server_negotiation.cc:407] Connection header received
0927 14:58:32.382268 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:58:32.382272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:58:32.382321 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:58:32.382391 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:58:32.382813 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:58:32.383522 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:58:32.386999 (+ 3477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:58:32.387672 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:58:32.387763 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:58:32.387841 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 14:58:32.388347 (+ 506us) server_negotiation.cc:299] Negotiation successful
0927 14:58:32.388446 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:58:32.390269 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:58:32.381243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45101 (local address 127.0.98.2:55455)
0927 14:58:32.382034 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0927 14:58:32.382049 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:58:32.382172 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:58:32.382517 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:58:32.382520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:58:32.382532 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:58:32.382659 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:58:32.382665 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:58:32.383666 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:58:32.383669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:58:32.384967 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:58:32.384981 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:58:32.388073 (+ 3092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:58:32.388076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:58:32.388192 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:58:32.388219 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 14:58:32.390002 (+ 1783us) client_negotiation.cc:240] Negotiation successful
0927 14:58:32.390095 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":126,"threads_started":1}
I20260927 14:58:32.391253 11771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82139f305e4147b88698289688314a01" candidate_uuid: "d01b4849a8244207a214ad55e53895bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d93bf0b8401424885bef354966cef6b" is_pre_election: true
I20260927 14:58:32.391594 11771 raft_consensus.cc:2454] T 82139f305e4147b88698289688314a01 P 7d93bf0b8401424885bef354966cef6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d01b4849a8244207a214ad55e53895bd in term 0.
I20260927 14:58:32.392284 11843 leader_election.cc:302] T 82139f305e4147b88698289688314a01 P d01b4849a8244207a214ad55e53895bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d93bf0b8401424885bef354966cef6b, d01b4849a8244207a214ad55e53895bd; no voters:
I20260927 14:58:32.392468 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:58:32.381823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59569 (local address 127.0.98.3:36747)
0927 14:58:32.383435 (+ 1612us) server_negotiation.cc:206] Beginning negotiation
0927 14:58:32.383440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:58:32.383453 (+ 13us) server_negotiation.cc:407] Connection header received
0927 14:58:32.383502 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:58:32.383505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:58:32.383549 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:58:32.383625 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:58:32.385447 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:58:32.386187 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:58:32.391056 (+ 4869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:58:32.391698 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:58:32.391789 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:58:32.391854 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 14:58:32.392247 (+ 393us) server_negotiation.cc:299] Negotiation successful
0927 14:58:32.392346 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1519}
I20260927 14:58:32.392665 12113 raft_consensus.cc:2781] T 82139f305e4147b88698289688314a01 P d01b4849a8244207a214ad55e53895bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:58:32.392766 12113 raft_consensus.cc:482] T 82139f305e4147b88698289688314a01 P d01b4849a8244207a214ad55e53895bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:58:32.392807 12113 raft_consensus.cc:3037] T 82139f305e4147b88698289688314a01 P d01b4849a8244207a214ad55e53895bd [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:58:32.394227 12113 raft_consensus.cc:504] T 82139f305e4147b88698289688314a01 P d01b4849a8244207a214ad55e53895bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d93bf0b8401424885bef354966cef6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45101 } } peers { permanent_uuid: "d01b4849a8244207a214ad55e53895bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41907 } } peers { permanent_uuid: "a048bdb8c3254c83b32dd5cab052c41e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 } }
I20260927 14:58:32.394687 12113 leader_election.cc:288] T 82139f305e4147b88698289688314a01 P d01b4849a8244207a214ad55e53895bd [CANDIDATE]: Term 1 election: Requested vote from peers 7d93bf0b8401424885bef354966cef6b (127.0.98.1:45101), a048bdb8c3254c83b32dd5cab052c41e (127.0.98.3:36747)
I20260927 14:58:32.395717 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:58:32.381453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36747 (local address 127.0.98.2:59569)
0927 14:58:32.382355 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0927 14:58:32.382367 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:58:32.382433 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:58:32.385129 (+ 2696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:58:32.385133 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:58:32.385159 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:58:32.385294 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:58:32.385301 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:58:32.386364 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:58:32.386368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:58:32.389872 (+ 3504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:58:32.389885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:58:32.391955 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:58:32.391959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:58:32.392087 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:58:32.392111 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 14:58:32.395436 (+ 3325us) client_negotiation.cc:240] Negotiation successful
0927 14:58:32.395544 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"thread_start_us":94,"threads_started":1}
I20260927 14:58:32.395702 11771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82139f305e4147b88698289688314a01" candidate_uuid: "d01b4849a8244207a214ad55e53895bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d93bf0b8401424885bef354966cef6b"
I20260927 14:58:32.395867 11771 raft_consensus.cc:3037] T 82139f305e4147b88698289688314a01 P 7d93bf0b8401424885bef354966cef6b [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:58:32.396693 12026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82139f305e4147b88698289688314a01" candidate_uuid: "d01b4849a8244207a214ad55e53895bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a048bdb8c3254c83b32dd5cab052c41e" is_pre_election: true
I20260927 14:58:32.397066 12026 raft_consensus.cc:2454] T 82139f305e4147b88698289688314a01 P a048bdb8c3254c83b32dd5cab052c41e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d01b4849a8244207a214ad55e53895bd in term 0.
I20260927 14:58:32.397217 11771 raft_consensus.cc:2454] T 82139f305e4147b88698289688314a01 P 7d93bf0b8401424885bef354966cef6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d01b4849a8244207a214ad55e53895bd in term 1.
I20260927 14:58:32.397440 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82139f305e4147b88698289688314a01" candidate_uuid: "d01b4849a8244207a214ad55e53895bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a048bdb8c3254c83b32dd5cab052c41e"
I20260927 14:58:32.397573 12039 raft_consensus.cc:3037] T 82139f305e4147b88698289688314a01 P a048bdb8c3254c83b32dd5cab052c41e [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:58:32.397606 11843 leader_election.cc:302] T 82139f305e4147b88698289688314a01 P d01b4849a8244207a214ad55e53895bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d93bf0b8401424885bef354966cef6b, d01b4849a8244207a214ad55e53895bd; no voters:
I20260927 14:58:32.398068 12113 raft_consensus.cc:2781] T 82139f305e4147b88698289688314a01 P d01b4849a8244207a214ad55e53895bd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:58:32.398280 12113 raft_consensus.cc:686] T 82139f305e4147b88698289688314a01 P d01b4849a8244207a214ad55e53895bd [term 1 LEADER]: Becoming Leader. State: Replica: d01b4849a8244207a214ad55e53895bd, State: Running, Role: LEADER
I20260927 14:58:32.398653 12113 consensus_queue.cc:237] T 82139f305e4147b88698289688314a01 P d01b4849a8244207a214ad55e53895bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d93bf0b8401424885bef354966cef6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45101 } } peers { permanent_uuid: "d01b4849a8244207a214ad55e53895bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41907 } } peers { permanent_uuid: "a048bdb8c3254c83b32dd5cab052c41e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 } }
I20260927 14:58:32.398931 12039 raft_consensus.cc:2454] T 82139f305e4147b88698289688314a01 P a048bdb8c3254c83b32dd5cab052c41e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d01b4849a8244207a214ad55e53895bd in term 1.
I20260927 14:58:32.402212 11603 catalog_manager.cc:5495] T 82139f305e4147b88698289688314a01 P d01b4849a8244207a214ad55e53895bd reported cstate change: term changed from 0 to 1, leader changed from <none> to d01b4849a8244207a214ad55e53895bd (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d01b4849a8244207a214ad55e53895bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d93bf0b8401424885bef354966cef6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d01b4849a8244207a214ad55e53895bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a048bdb8c3254c83b32dd5cab052c41e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 } health_report { overall_health: UNKNOWN } } }
I20260927 14:58:32.436920 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:58:32.432287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43612 (local address 127.0.98.2:41907)
0927 14:58:32.432413 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 14:58:32.432419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:58:32.432459 (+ 40us) server_negotiation.cc:407] Connection header received
0927 14:58:32.432649 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:58:32.432652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:58:32.432720 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:58:32.432838 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:58:32.433110 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:58:32.433856 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:58:32.435130 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:58:32.435439 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:58:32.435541 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:58:32.435895 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:58:32.436486 (+ 591us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:58:32.436605 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 14:58:32.436699 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 14:58:32.436768 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 14:58:32.444584 12039 raft_consensus.cc:1261] T 82139f305e4147b88698289688314a01 P a048bdb8c3254c83b32dd5cab052c41e [term 1 FOLLOWER]: Refusing update from remote peer d01b4849a8244207a214ad55e53895bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:58:32.444711 11771 raft_consensus.cc:1261] T 82139f305e4147b88698289688314a01 P 7d93bf0b8401424885bef354966cef6b [term 1 FOLLOWER]: Refusing update from remote peer d01b4849a8244207a214ad55e53895bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:58:32.445451 12113 consensus_queue.cc:1035] T 82139f305e4147b88698289688314a01 P d01b4849a8244207a214ad55e53895bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d93bf0b8401424885bef354966cef6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:58:32.445930 12117 consensus_queue.cc:1035] T 82139f305e4147b88698289688314a01 P d01b4849a8244207a214ad55e53895bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a048bdb8c3254c83b32dd5cab052c41e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:58:32.461148 12122 mvcc.cc:205] Tried to move back new op lower bound from 7333974476564336640 to 7333974476388995072. Current Snapshot: MvccSnapshot[applied={T|T < 7333974476564336640}]
I20260927 14:58:32.466079 12121 mvcc.cc:205] Tried to move back new op lower bound from 7333974476564336640 to 7333974476388995072. Current Snapshot: MvccSnapshot[applied={T|T < 7333974476564336640}]
I20260927 14:58:32.472194 12120 mvcc.cc:205] Tried to move back new op lower bound from 7333974476564336640 to 7333974476388995072. Current Snapshot: MvccSnapshot[applied={T|T < 7333974476564336640 or (T in {7333974476564336640})}]
I20260927 14:58:32.477316 11603 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:58:32.477519 11603 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:58:32.480837 11603 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b7ce625274f4b4ca3e109b33b7c8861: Sending DeleteTablet for 3 replicas of tablet 82139f305e4147b88698289688314a01
I20260927 14:58:32.481873 11742 tablet_service.cc:1514] Processing DeleteTablet for tablet 82139f305e4147b88698289688314a01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:58:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41946
I20260927 14:58:32.482772 11872 tablet_service.cc:1514] Processing DeleteTablet for tablet 82139f305e4147b88698289688314a01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:58:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43602
I20260927 14:58:32.483191 12021 tablet_service.cc:1514] Processing DeleteTablet for tablet 82139f305e4147b88698289688314a01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:58:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33066
I20260927 14:58:32.484436 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 11684
I20260927 14:58:32.489573 12131 tablet_replica.cc:330] T 82139f305e4147b88698289688314a01 P a048bdb8c3254c83b32dd5cab052c41e: stopping tablet replica
I20260927 14:58:32.493631 12131 raft_consensus.cc:2229] T 82139f305e4147b88698289688314a01 P a048bdb8c3254c83b32dd5cab052c41e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:58:32.494028 12131 raft_consensus.cc:2258] T 82139f305e4147b88698289688314a01 P a048bdb8c3254c83b32dd5cab052c41e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:58:32.495890 12131 ts_tablet_manager.cc:1905] T 82139f305e4147b88698289688314a01 P a048bdb8c3254c83b32dd5cab052c41e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:58:32.496259 12130 tablet_replica.cc:330] T 82139f305e4147b88698289688314a01 P d01b4849a8244207a214ad55e53895bd: stopping tablet replica
I20260927 14:58:32.496663 12130 raft_consensus.cc:2229] T 82139f305e4147b88698289688314a01 P d01b4849a8244207a214ad55e53895bd [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:58:32.499400 12130 raft_consensus.cc:2258] T 82139f305e4147b88698289688314a01 P d01b4849a8244207a214ad55e53895bd [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 14:58:32.499825 11590 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45101 (error 108)
I20260927 14:58:32.501048 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 11823
I20260927 14:58:32.503358 12131 ts_tablet_manager.cc:1918] T 82139f305e4147b88698289688314a01 P a048bdb8c3254c83b32dd5cab052c41e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:58:32.503597 12131 log.cc:1196] T 82139f305e4147b88698289688314a01 P a048bdb8c3254c83b32dd5cab052c41e: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790521004237705-392-0/minicluster-data/ts-2/wal/wals/82139f305e4147b88698289688314a01
I20260927 14:58:32.503985 12131 ts_tablet_manager.cc:1939] T 82139f305e4147b88698289688314a01 P a048bdb8c3254c83b32dd5cab052c41e: Deleting consensus metadata
I20260927 14:58:32.505066 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:58:32.504757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45101 (local address 127.0.0.1:41956)
0927 14:58:32.504888 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 14:58:32.504969 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 14:58:32.505407 11590 catalog_manager.cc:4584] TS 7d93bf0b8401424885bef354966cef6b (127.0.98.1:45101): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 82139f305e4147b88698289688314a01: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45101: connect: Connection refused (error 111)
I20260927 14:58:32.506361 11589 catalog_manager.cc:4841] TS a048bdb8c3254c83b32dd5cab052c41e (127.0.98.3:36747): tablet 82139f305e4147b88698289688314a01 (table test-table [id=c4634a1c569041a0a6092a6d1ec3a283]) successfully deleted
I20260927 14:58:32.509119 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:58:32.508785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41907 (local address 127.0.0.1:43622)
0927 14:58:32.508969 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 14:58:32.509021 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260927 14:58:32.509620 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 11958
I20260927 14:58:32.517329 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 11571
I20260927 14:58:32.527199 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 14:58:32.862434 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 14:58:32.862576 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 14:58:33.198141 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 14:58:33.198 UTC [11157] LOG: received smart shutdown request
2026-09-27 14:58:33.201 UTC [11157] LOG: background worker "logical replication launcher" (PID 11166) exited with exit code 1
2026-09-27 14:58:33.201 UTC [11161] LOG: shutting down
2026-09-27 14:58:33.358 UTC [11157] LOG: database system is shut down
2026-09-27T14:58:33Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48029 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:58:33 dist-test-slave-pz0h krb5kdc[12142](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:58:33 dist-test-slave-pz0h krb5kdc[12142](info): set up 2 sockets
Sep 27 14:58:33 dist-test-slave-pz0h krb5kdc[12142](info): commencing operation
krb5kdc: starting...
W20260927 14:58:34.406677 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 14:58:34 dist-test-slave-pz0h krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:58:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:58:34Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 14:58:34.565881 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskDguobN/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/postgres -l logfile start
2026-09-27 14:58:36.939 UTC [12172] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 14:58:36.939 UTC [12172] LOG: listening on IPv4 address "127.0.98.20", port 45501
2026-09-27 14:58:36.943 UTC [12172] LOG: listening on Unix socket "/tmp/.s.PGSQL.45501"
2026-09-27 14:58:36.970 UTC [12175] LOG: database system was shut down at 2026-09-27 14:58:35 UTC
2026-09-27 14:58:36.974 UTC [12172] LOG: database system is ready to accept connections
I20260927 14:58:37.842453 392 mini_postgres.cc:96] Postgres bound to 45501
2026-09-27 14:58:37.848 UTC [12183] FATAL: database "slave" does not exist
127.0.98.20:45501 - accepting connections
I20260927 14:58:37.848886 392 mini_ranger.cc:161] Starting Ranger...
I20260927 14:58:37.864715 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 14:58:38.305356 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 14:58:38.305439 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ranger-admin
2026-09-27 14:58:38,588 [I] DB FLAVOR :POSTGRES
2026-09-27 14:58:38,588 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 14:58:38,588 [I] Checking connection..
2026-09-27 14:58:38,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 14:58:38,829 [I] Checking connection passed.
2026-09-27 14:58:38,829 [I] --------- Verifying version history table ---------
2026-09-27 14:58:38,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 14:58:39,097 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 14:58:39,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 14:58:39,382 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 14:58:39,382 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 14:58:39,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 14:58:39.627 UTC [12247] WARNING: there is no transaction in progress
2026-09-27 14:58:39,633 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 14:58:39,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 14:58:39,886 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 14:58:39,886 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 14:58:39,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 14:58:40,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 14:58:40,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-pz0h.c.gcp-upstream.internal','N') ;"
2026-09-27 14:58:40,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 14:58:40,881 [I] Table x_portal_user does not exist in database ranger
2026-09-27 14:58:40,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 14:58:41,145 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 14:58:41,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 14:58:41,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 14:58:41,661 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 14:58:41,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 14:58:42.661 UTC [12355] WARNING: there is no transaction in progress
2026-09-27 14:58:42.677 UTC [12355] WARNING: there is no transaction in progress
2026-09-27 14:58:42.694 UTC [12355] WARNING: there is no transaction in progress
2026-09-27 14:58:42.710 UTC [12355] WARNING: there is no transaction in progress
2026-09-27 14:58:42.727 UTC [12355] WARNING: there is no transaction in progress
2026-09-27 14:58:42.745 UTC [12355] WARNING: there is no transaction in progress
2026-09-27 14:58:42.803 UTC [12355] WARNING: there is no transaction in progress
2026-09-27 14:58:42.811 UTC [12355] WARNING: there is no transaction in progress
2026-09-27 14:58:42.826 UTC [12355] WARNING: there is no transaction in progress
2026-09-27 14:58:42.837 UTC [12355] WARNING: there is no transaction in progress
2026-09-27 14:58:42.849 UTC [12355] WARNING: there is no transaction in progress
2026-09-27 14:58:42.865 UTC [12355] WARNING: there is no transaction in progress
2026-09-27 14:58:42.875 UTC [12355] WARNING: there is no transaction in progress
2026-09-27 14:58:42.882 UTC [12355] WARNING: there is no transaction in progress
2026-09-27 14:58:42.890 UTC [12355] WARNING: there is no transaction in progress
2026-09-27 14:58:42.897 UTC [12355] WARNING: there is no transaction in progress
2026-09-27 14:58:43,618 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 14:58:43,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 14:58:43,885 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 14:58:43,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 14:58:44,152 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 14:58:44,152 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 14:58:44,417 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 14:58:44,417 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 14:58:44,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 14:58:44,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-pz0h.c.gcp-upstream.internal';"
2026-09-27 14:58:45,187 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 14:58:45,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 14:58:45,455 [I] DB_PATCHES have already been applied
I20260927 14:58:45.460939 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskDguobN/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 14:58:45.461015 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 14:58:45.464497 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:42475
Sep 27, 2026 2:58:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:58:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:58:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 2:58:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 2:58:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 2:58:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 2:58:46 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 14:58:46.456058 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.607s user 0.000s sys 0.009s
Sep 27, 2026 2:59:08 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 2:59:08 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 2:59:08 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 2:59:08 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 2:59:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 2:59:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 2:59:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 2:59:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 2:59:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 2:59:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 2:59:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 2:59:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 2:59:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 2:59:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 2:59:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 2:59:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 2:59:08 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 2:59:09 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 14:59:09.749084 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 14:59:09 dist-test-slave-pz0h krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521149, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 14:59:09.850057 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 14:59:09.862085 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 14:59:10.611810 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 14:59:10.611891 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ranger-kms
I20260927 14:59:10.611910 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:45501
2026-09-27 14:59:10,766 [I] DB FLAVOR :POSTGRES
2026-09-27 14:59:10,766 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 14:59:10,766 [I] Checking connection
2026-09-27 14:59:10,766 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45501/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 14:59:11,012 [I] connection success
2026-09-27 14:59:11,013 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 14:59:11,013 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 14:59:11,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45501/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 14:59:11,260 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 14:59:11,261 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 14:59:11,261 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 14:59:11,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45501/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 14:59:11,520 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 14:59:12.058246 392 mini_ranger_kms.cc:310] Created kms service
I20260927 14:59:12.274837 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 14:59:12.319430 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 14:59:12 dist-test-slave-pz0h krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521149, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:59:12.750483 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 14:59:12.750629 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskDguobN/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskDguobN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 14:59:12.750646 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 14:59:12.754552 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12536
I20260927 14:59:12.754637 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:38699
Sep 27, 2026 2:59:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:59:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:59:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 2:59:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 2:59:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 2:59:14 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 14:59:14.802346 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.952s user 0.013s sys 0.125s
I20260927 14:59:14.802477 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 14:59:14.802538 392 mini_ranger_kms.cc:401] 127.0.98.20:38699/kms/v1/keys
Sep 27 14:59:14 dist-test-slave-pz0h krb5kdc[12142](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790521154, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 14:59:15 dist-test-slave-pz0h krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521154, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 14:59:15 dist-test-slave-pz0h krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521154, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 14:59:18 dist-test-slave-pz0h krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:59:18.360538 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37785
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37785
--ranger_config_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38699/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:59:18.473730 12586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:59:18.474056 12586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:59:18.474112 12586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:59:18.478384 12586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:59:18.478468 12586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:59:18.478497 12586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:59:18.478518 12586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:59:18.478536 12586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:59:18.484434 12586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:38699/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37785
--ranger_config_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:59:18.485585 12586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:59:18.486640 12586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:59:18.495703 12592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:18.495752 12594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:18.495937 12591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:18.495911 12586 server_base.cc:1034] running on GCE node
I20260927 14:59:18.496603 12586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:59:18.497776 12586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:59:18.498898 12586 hybrid_clock.cc:648] HybridClock initialized: now 1790521158498868 us; error 48 us; skew 500 ppm
Sep 27 14:59:18 dist-test-slave-pz0h krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:59:18.509191 12586 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:59:18.510325 12586 webserver.cc:460] Webserver started at http://127.0.98.62:35057/ using document root <none> and password file <none>
I20260927 14:59:18.510967 12586 fs_manager.cc:359] Metadata directory not provided
I20260927 14:59:18.511039 12586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:59:18.511248 12586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 14:59:18 dist-test-slave-pz0h krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:59:18.575261 12586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "4818278912b049b787bd28c7aca1c312"
format_stamp: "Formatted at 2026-09-27 14:59:18 on dist-test-slave-pz0h"
server_key: "b647d0009d3e6c67ecb62e5f490a32ad"
server_key_iv: "119db1254af2c5f4c561cd1c180ddda8"
server_key_version: "kuduclusterkey@0"
I20260927 14:59:18.575912 12586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "4818278912b049b787bd28c7aca1c312"
format_stamp: "Formatted at 2026-09-27 14:59:18 on dist-test-slave-pz0h"
server_key: "b647d0009d3e6c67ecb62e5f490a32ad"
server_key_iv: "119db1254af2c5f4c561cd1c180ddda8"
server_key_version: "kuduclusterkey@0"
I20260927 14:59:18.580207 12586 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 14:59:18.598500 12601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:18.599933 12586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:59:18.600065 12586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "4818278912b049b787bd28c7aca1c312"
format_stamp: "Formatted at 2026-09-27 14:59:18 on dist-test-slave-pz0h"
server_key: "b647d0009d3e6c67ecb62e5f490a32ad"
server_key_iv: "119db1254af2c5f4c561cd1c180ddda8"
server_key_version: "kuduclusterkey@0"
I20260927 14:59:18.600214 12586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:59:18.630403 12586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:59:18.633461 12586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:59:18.633659 12586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:59:18.641178 12586 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37785
I20260927 14:59:18.641206 12653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37785 every 8 connection(s)
I20260927 14:59:18.642222 12586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:59:18.645882 12654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:59:18.649547 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 12586
I20260927 14:59:18.649669 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 14:59:18.649928 392 external_mini_cluster.cc:1428] Setting key 9c6dfa2ab714464dc69c047563201887
I20260927 14:59:18.653460 12654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312: Bootstrap starting.
I20260927 14:59:18.656414 12654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312: Neither blocks nor log segments found. Creating new log.
I20260927 14:59:18.657322 12654 log.cc:824] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312: Log is configured to *not* fsync() on all Append() calls
Sep 27 14:59:18 dist-test-slave-pz0h krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:59:18.659840 12654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312: No bootstrap required, opened a new log
I20260927 14:59:18.664026 12654 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4818278912b049b787bd28c7aca1c312" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37785 } }
I20260927 14:59:18.664361 12654 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:59:18.664445 12654 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4818278912b049b787bd28c7aca1c312, State: Initialized, Role: FOLLOWER
I20260927 14:59:18.664626 12657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:18.651417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33380 (local address 127.0.98.62:37785)
0927 14:59:18.651855 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:18.651872 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:18.651909 (+ 37us) server_negotiation.cc:407] Connection header received
0927 14:59:18.652574 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:18.652592 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:18.652949 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:18.653286 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:18.654071 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:18.655130 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:18.656007 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:18.656498 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:18.656654 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:18.659292 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:18.659298 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:18.659314 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:18.659344 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:18.661495 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:18.661954 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:18.661960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:18.661965 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:18.662084 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:18.662349 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:18.662352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:18.662353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:18.662775 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:18.663011 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:18.663294 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 14:59:18.663740 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":130,"threads_started":1}
I20260927 14:59:18.664958 12654 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4818278912b049b787bd28c7aca1c312" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37785 } }
I20260927 14:59:18.665127 12654 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:59:18.665211 12654 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:59:18.665305 12654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:59:18.666510 12654 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4818278912b049b787bd28c7aca1c312" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37785 } }
I20260927 14:59:18.666919 12654 leader_election.cc:302] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4818278912b049b787bd28c7aca1c312; no voters:
I20260927 14:59:18.667367 12654 leader_election.cc:288] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:59:18.667461 12659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:59:18.667686 12659 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [term 1 LEADER]: Becoming Leader. State: Replica: 4818278912b049b787bd28c7aca1c312, State: Running, Role: LEADER
I20260927 14:59:18.668032 12659 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4818278912b049b787bd28c7aca1c312" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37785 } }
I20260927 14:59:18.668665 12654 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:59:18.670383 12660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4818278912b049b787bd28c7aca1c312" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4818278912b049b787bd28c7aca1c312" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37785 } } }
I20260927 14:59:18.670383 12661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4818278912b049b787bd28c7aca1c312. Latest consensus state: current_term: 1 leader_uuid: "4818278912b049b787bd28c7aca1c312" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4818278912b049b787bd28c7aca1c312" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37785 } } }
I20260927 14:59:18.670536 12660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [sys.catalog]: This master's current role is: LEADER
I20260927 14:59:18.670536 12661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [sys.catalog]: This master's current role is: LEADER
I20260927 14:59:18.670857 12654 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 14:59:18.672257 12673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:59:18.679028 12673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:59:18.686344 12673 catalog_manager.cc:1347] Generated new cluster ID: 89aeb2e414164b15af0578d45d300885
I20260927 14:59:18.686558 12673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:59:18.707541 12673 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:59:18.708920 12673 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:59:18.717767 12673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312: Generated new TSK 0
I20260927 14:59:18.718954 12673 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:59:19.437392 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38699/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37785
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:59:19.569970 12699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:59:19.570387 12699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:59:19.570457 12699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:59:19.576890 12699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:59:19.576992 12699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:59:19.577090 12699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:59:19.585579 12699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:38699/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:59:19.587534 12699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:59:19.588961 12699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:59:19.600338 12705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:19.600338 12704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:19.600338 12707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:19.600867 12699 server_base.cc:1034] running on GCE node
I20260927 14:59:19.601323 12699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:59:19.602043 12699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:59:19.603250 12699 hybrid_clock.cc:648] HybridClock initialized: now 1790521159603222 us; error 47 us; skew 500 ppm
Sep 27 14:59:19 dist-test-slave-pz0h krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:59:19.606290 12699 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:59:19.607421 12699 webserver.cc:460] Webserver started at http://127.0.98.1:37887/ using document root <none> and password file <none>
I20260927 14:59:19.608040 12699 fs_manager.cc:359] Metadata directory not provided
I20260927 14:59:19.608114 12699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:59:19.608330 12699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 14:59:19 dist-test-slave-pz0h krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:59:19.624022 12699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/data/instance:
uuid: "b45bb6d0a0904f2cb868bdcd89d7d5ed"
format_stamp: "Formatted at 2026-09-27 14:59:19 on dist-test-slave-pz0h"
server_key: "9ac4569d1b6d09101ff31a5a5b488d00"
server_key_iv: "7b36e760fdeabf3651f5589e8a1d0a2c"
server_key_version: "kuduclusterkey@0"
I20260927 14:59:19.624648 12699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b45bb6d0a0904f2cb868bdcd89d7d5ed"
format_stamp: "Formatted at 2026-09-27 14:59:19 on dist-test-slave-pz0h"
server_key: "9ac4569d1b6d09101ff31a5a5b488d00"
server_key_iv: "7b36e760fdeabf3651f5589e8a1d0a2c"
server_key_version: "kuduclusterkey@0"
I20260927 14:59:19.630174 12699 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 14:59:19.662622 12716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:19.664541 12699 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.004s sys 0.000s
I20260927 14:59:19.664729 12699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "b45bb6d0a0904f2cb868bdcd89d7d5ed"
format_stamp: "Formatted at 2026-09-27 14:59:19 on dist-test-slave-pz0h"
server_key: "9ac4569d1b6d09101ff31a5a5b488d00"
server_key_iv: "7b36e760fdeabf3651f5589e8a1d0a2c"
server_key_version: "kuduclusterkey@0"
I20260927 14:59:19.664911 12699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:59:19.685891 12699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:59:19.689911 12699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:59:19.690135 12699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:59:19.690917 12699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:59:19.692044 12699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:59:19.692109 12699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:19.692163 12699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:59:19.692191 12699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:19.706708 12699 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40761
I20260927 14:59:19.708043 12699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:59:19.713562 12829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40761 every 8 connection(s)
I20260927 14:59:19.717911 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 12699
I20260927 14:59:19.718034 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:59:19.718340 392 external_mini_cluster.cc:1428] Setting key b0ee7cb73147233a35d930707162a72a
Sep 27 14:59:19 dist-test-slave-pz0h krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:59:19.729290 12832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:19.711028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38229 (local address 127.0.98.62:37785)
0927 14:59:19.714165 (+ 3137us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:19.714175 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:19.716051 (+ 1876us) server_negotiation.cc:407] Connection header received
0927 14:59:19.716111 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:19.716114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:19.716180 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:19.716260 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:19.717978 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:19.718911 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:19.720700 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:19.721064 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:19.721145 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:19.724477 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:19.724481 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:19.724484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:19.724518 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:19.726398 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:19.727234 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:19.727238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:19.727241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:19.727345 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:19.727815 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:19.727818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:19.727820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:19.728070 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:19.728153 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:19.728938 (+ 785us) server_negotiation.cc:299] Negotiation successful
0927 14:59:19.729096 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3031,"thread_start_us":129,"threads_started":1}
I20260927 14:59:19.731475 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:19.712037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37785 (local address 127.0.98.1:38229)
0927 14:59:19.714451 (+ 2414us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:19.714516 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:19.715689 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:19.716624 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:19.716635 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:19.717038 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:59:19.717772 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:19.717794 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:19.719080 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:19.719084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:19.720531 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:19.720545 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:19.721372 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:19.721376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:19.721591 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:19.722351 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:59:19.722372 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:59:19.724223 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:59:19.726586 (+ 2363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:19.726593 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:19.726630 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:19.727085 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:19.727467 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:19.727470 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:19.727472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:19.727698 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:19.728223 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:59:19.728231 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:59:19.728632 (+ 401us) client_negotiation.cc:769] Sending connection context
0927 14:59:19.729844 (+ 1212us) client_negotiation.cc:240] Negotiation successful
0927 14:59:19.730261 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2174,"thread_start_us":156,"threads_started":1}
I20260927 14:59:19.733534 12830 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37785
I20260927 14:59:19.734163 12830 heartbeater.cc:464] Registering TS with master...
I20260927 14:59:19.735069 12830 heartbeater.cc:511] Master 127.0.98.62:37785 requested a full tablet report, sending...
I20260927 14:59:19.737295 12617 ts_manager.cc:194] Registered new tserver with Master: b45bb6d0a0904f2cb868bdcd89d7d5ed (127.0.98.1:40761)
I20260927 14:59:19.739121 12617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38229
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:59:19.794147 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38699/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37785
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 14:59:19 dist-test-slave-pz0h krb5kdc[12142](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790521159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 14:59:19.972956 12838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:59:19.973444 12838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:59:19.973557 12838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:59:19.979447 12838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:59:19.979656 12838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:59:19.979823 12838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:59:19.986855 12838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:38699/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:59:19.988035 12838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:59:19.989073 12838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:59:20.006193 12844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:20.006779 12838 server_base.cc:1034] running on GCE node
W20260927 14:59:20.006778 12845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:20.007220 12847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:20.007776 12838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:59:20.008658 12838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:59:20.009832 12838 hybrid_clock.cc:648] HybridClock initialized: now 1790521160009769 us; error 85 us; skew 500 ppm
Sep 27 14:59:20 dist-test-slave-pz0h krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:59:20.013783 12838 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:59:20.015280 12838 webserver.cc:460] Webserver started at http://127.0.98.2:34463/ using document root <none> and password file <none>
I20260927 14:59:20.016185 12838 fs_manager.cc:359] Metadata directory not provided
I20260927 14:59:20.016364 12838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:59:20.016737 12838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 14:59:20 dist-test-slave-pz0h krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:59:20.044314 12838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/data/instance:
uuid: "9538f91af0ec43fd9c80cdb5d8cf0300"
format_stamp: "Formatted at 2026-09-27 14:59:20 on dist-test-slave-pz0h"
server_key: "a22f87b5ac03211fbd977eb5ae5f6a41"
server_key_iv: "5d8abcb73dc21933e2f522635e870391"
server_key_version: "kuduclusterkey@0"
I20260927 14:59:20.045367 12838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9538f91af0ec43fd9c80cdb5d8cf0300"
format_stamp: "Formatted at 2026-09-27 14:59:20 on dist-test-slave-pz0h"
server_key: "a22f87b5ac03211fbd977eb5ae5f6a41"
server_key_iv: "5d8abcb73dc21933e2f522635e870391"
server_key_version: "kuduclusterkey@0"
I20260927 14:59:20.051381 12838 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 14:59:20.079662 12854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:20.081385 12838 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 14:59:20.081547 12838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "9538f91af0ec43fd9c80cdb5d8cf0300"
format_stamp: "Formatted at 2026-09-27 14:59:20 on dist-test-slave-pz0h"
server_key: "a22f87b5ac03211fbd977eb5ae5f6a41"
server_key_iv: "5d8abcb73dc21933e2f522635e870391"
server_key_version: "kuduclusterkey@0"
I20260927 14:59:20.081707 12838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:59:20.109609 12838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:59:20.113735 12838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:59:20.114073 12838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:59:20.114877 12838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:59:20.116029 12838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:59:20.116139 12838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:20.116248 12838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:59:20.116312 12838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:20.131024 12838 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46099
I20260927 14:59:20.132417 12838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:59:20.133970 12967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46099 every 8 connection(s)
I20260927 14:59:20.140910 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 12838
I20260927 14:59:20.141058 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:59:20.141374 392 external_mini_cluster.cc:1428] Setting key 8805ad9f86290b3597bd549f8475406b
Sep 27 14:59:20 dist-test-slave-pz0h krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:59:20.154413 12832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:20.138282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38883 (local address 127.0.98.62:37785)
0927 14:59:20.138419 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:20.138425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:20.139346 (+ 921us) server_negotiation.cc:407] Connection header received
0927 14:59:20.140759 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:20.140764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:20.140826 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:20.140883 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:20.143807 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:20.144555 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:20.146312 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:20.146679 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:20.146753 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:20.150043 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:20.150047 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:20.150050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:20.150087 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:20.151819 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:20.152516 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:20.152520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:20.152522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:20.152611 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:20.153024 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:20.153027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:20.153028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:20.153266 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:20.153359 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:20.154069 (+ 710us) server_negotiation.cc:299] Negotiation successful
0927 14:59:20.154221 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 14:59:20.155596 12970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:20.136912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37785 (local address 127.0.98.2:38883)
0927 14:59:20.139136 (+ 2224us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:20.139209 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:20.140444 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:20.141558 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:20.141570 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:20.142787 (+ 1217us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:59:20.143619 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:20.143642 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:20.144736 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:20.144740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:20.146154 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:20.146167 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:20.147026 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:20.147031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:20.147256 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:20.148001 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:59:20.148023 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:59:20.149796 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:59:20.151987 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:20.151995 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:20.152014 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:20.152391 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:20.152707 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:20.152710 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:20.152713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:20.152911 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:20.153362 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:59:20.153369 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:59:20.153756 (+ 387us) client_negotiation.cc:769] Sending connection context
0927 14:59:20.153972 (+ 216us) client_negotiation.cc:240] Negotiation successful
0927 14:59:20.154338 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1949,"thread_start_us":183,"threads_started":1}
I20260927 14:59:20.156975 12968 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37785
I20260927 14:59:20.157251 12968 heartbeater.cc:464] Registering TS with master...
I20260927 14:59:20.157869 12968 heartbeater.cc:511] Master 127.0.98.62:37785 requested a full tablet report, sending...
I20260927 14:59:20.159173 12617 ts_manager.cc:194] Registered new tserver with Master: 9538f91af0ec43fd9c80cdb5d8cf0300 (127.0.98.2:46099)
I20260927 14:59:20.159986 12617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38883
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:59:20.182120 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38699/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37785
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:59:20.353286 12973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:59:20.353762 12973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:59:20.353866 12973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:59:20.360100 12973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:59:20.360280 12973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:59:20.360428 12973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:59:20.369728 12973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:38699/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:59:20.371474 12973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:59:20.372933 12973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 27 14:59:20 dist-test-slave-pz0h krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 14:59:20.385759 12979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:20.386785 12978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:20.387430 12973 server_base.cc:1034] running on GCE node
W20260927 14:59:20.387609 12981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:20.388124 12973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:59:20.388883 12973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:59:20.393713 12973 hybrid_clock.cc:648] HybridClock initialized: now 1790521160393607 us; error 91 us; skew 500 ppm
Sep 27 14:59:20 dist-test-slave-pz0h krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:59:20.398185 12973 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:59:20.399601 12973 webserver.cc:460] Webserver started at http://127.0.98.3:46607/ using document root <none> and password file <none>
I20260927 14:59:20.400349 12973 fs_manager.cc:359] Metadata directory not provided
I20260927 14:59:20.400444 12973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:59:20.400702 12973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 14:59:20 dist-test-slave-pz0h krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:59:20.421926 12973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/data/instance:
uuid: "a06c38d076704da3bd36b97d24e126df"
format_stamp: "Formatted at 2026-09-27 14:59:20 on dist-test-slave-pz0h"
server_key: "ee6cd30048ad63ecbcaeed6715de823d"
server_key_iv: "1c68a0b3b2d8badee0eebe2ac8a1ca96"
server_key_version: "kuduclusterkey@0"
I20260927 14:59:20.422915 12973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a06c38d076704da3bd36b97d24e126df"
format_stamp: "Formatted at 2026-09-27 14:59:20 on dist-test-slave-pz0h"
server_key: "ee6cd30048ad63ecbcaeed6715de823d"
server_key_iv: "1c68a0b3b2d8badee0eebe2ac8a1ca96"
server_key_version: "kuduclusterkey@0"
I20260927 14:59:20.440765 12973 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.004s sys 0.000s
I20260927 14:59:20.468164 12988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:20.470221 12973 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.003s sys 0.000s
I20260927 14:59:20.470405 12973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "a06c38d076704da3bd36b97d24e126df"
format_stamp: "Formatted at 2026-09-27 14:59:20 on dist-test-slave-pz0h"
server_key: "ee6cd30048ad63ecbcaeed6715de823d"
server_key_iv: "1c68a0b3b2d8badee0eebe2ac8a1ca96"
server_key_version: "kuduclusterkey@0"
I20260927 14:59:20.470564 12973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:59:20.514422 12973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:59:20.522024 12973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:59:20.522297 12973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:59:20.523157 12973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:59:20.524381 12973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:59:20.524497 12973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:20.524597 12973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:59:20.524657 12973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:20.535044 12973 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39971
I20260927 14:59:20.535230 13101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39971 every 8 connection(s)
I20260927 14:59:20.536353 12973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:59:20.542415 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 12973
I20260927 14:59:20.542537 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:59:20.542889 392 external_mini_cluster.cc:1428] Setting key c446f92a628749c69684c74d3ff4a817
Sep 27 14:59:20 dist-test-slave-pz0h krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:59:20.555219 12832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:20.540153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48747 (local address 127.0.98.62:37785)
0927 14:59:20.540313 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:20.540318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:20.540699 (+ 381us) server_negotiation.cc:407] Connection header received
0927 14:59:20.541987 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:20.541992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:20.542049 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:20.542099 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:20.544497 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:20.545320 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:20.547058 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:20.547411 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:20.547526 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:20.550507 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:20.550512 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:20.550514 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:20.550545 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:20.552350 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:20.553118 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:20.553122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:20.553124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:20.553237 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:20.553698 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:20.553702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:20.553704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:20.553957 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:20.554041 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:20.554872 (+ 831us) server_negotiation.cc:299] Negotiation successful
0927 14:59:20.555022 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 14:59:20.557690 13104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:20.538981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37785 (local address 127.0.98.3:48747)
0927 14:59:20.540510 (+ 1529us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:20.540576 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:20.541712 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:20.542553 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:20.542563 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:20.543554 (+ 991us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:59:20.544347 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:20.544371 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:20.545477 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:20.545481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:20.546899 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:20.546914 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:20.547589 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:20.547594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:20.547810 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:20.548549 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:59:20.548571 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:59:20.550275 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:59:20.552530 (+ 2255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:20.552537 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:20.552560 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:20.552967 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:20.553360 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:20.553363 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:20.553366 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:20.553576 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:20.554110 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:59:20.554118 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:59:20.554535 (+ 417us) client_negotiation.cc:769] Sending connection context
0927 14:59:20.556123 (+ 1588us) client_negotiation.cc:240] Negotiation successful
0927 14:59:20.556558 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1294,"thread_start_us":149,"threads_started":1}
I20260927 14:59:20.558993 13102 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37785
I20260927 14:59:20.559278 13102 heartbeater.cc:464] Registering TS with master...
I20260927 14:59:20.559904 13102 heartbeater.cc:511] Master 127.0.98.62:37785 requested a full tablet report, sending...
I20260927 14:59:20.561074 12617 ts_manager.cc:194] Registered new tserver with Master: a06c38d076704da3bd36b97d24e126df (127.0.98.3:39971)
I20260927 14:59:20.561872 12617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48747
I20260927 14:59:20.568914 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:59:20.569581 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 12699
I20260927 14:59:20.575248 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 12838
I20260927 14:59:20.581471 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 12973
I20260927 14:59:20.586582 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 12586
I20260927 14:59:20.594158 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37785
--webserver_interface=127.0.98.62
--webserver_port=35057
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37785
--ranger_config_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38699/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38699/kms with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:59:20.705073 13105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:59:20.705391 13105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:59:20.705433 13105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:59:20.709745 13105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:59:20.709817 13105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:59:20.709836 13105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:59:20.709851 13105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:59:20.709864 13105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:59:20.716605 13105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38699/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37785
--ranger_config_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:59:20.717826 13105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:59:20.718981 13105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:59:20.727933 13111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:20.727991 13105 server_base.cc:1034] running on GCE node
W20260927 14:59:20.727942 13110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:20.727942 13113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:20.728921 13105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:59:20.730185 13105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:59:20.731380 13105 hybrid_clock.cc:648] HybridClock initialized: now 1790521160731354 us; error 51 us; skew 500 ppm
Sep 27 14:59:20 dist-test-slave-pz0h krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:59:20.734624 13105 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:59:20.735756 13105 webserver.cc:460] Webserver started at http://127.0.98.62:35057/ using document root <none> and password file <none>
I20260927 14:59:20.736335 13105 fs_manager.cc:359] Metadata directory not provided
I20260927 14:59:20.736418 13105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 14:59:20 dist-test-slave-pz0h krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:59:20.799798 13105 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:59:20.802266 13121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:20.804045 13105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 14:59:20.804206 13105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "4818278912b049b787bd28c7aca1c312"
format_stamp: "Formatted at 2026-09-27 14:59:18 on dist-test-slave-pz0h"
server_key: "b647d0009d3e6c67ecb62e5f490a32ad"
server_key_iv: "119db1254af2c5f4c561cd1c180ddda8"
server_key_version: "kuduclusterkey@0"
I20260927 14:59:20.804806 13105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:59:20.818354 13105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:59:20.821084 13105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:59:20.821275 13105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:59:20.828447 13105 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37785
I20260927 14:59:20.828461 13173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37785 every 8 connection(s)
I20260927 14:59:20.829618 13105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:59:20.831673 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 13105
I20260927 14:59:20.832152 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38699/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40761
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37785
--webserver_port=37887
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38699/kms with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:59:20.834700 13174 sys_catalog.cc:263] Verifying existing consensus state
I20260927 14:59:20.836489 13174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312: Bootstrap starting.
I20260927 14:59:20.853168 13174 log.cc:824] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312: Log is configured to *not* fsync() on all Append() calls
I20260927 14:59:20.860141 13174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 14:59:20.860530 13174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312: Bootstrap complete.
I20260927 14:59:20.864588 13174 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4818278912b049b787bd28c7aca1c312" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37785 } }
I20260927 14:59:20.865015 13174 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4818278912b049b787bd28c7aca1c312, State: Initialized, Role: FOLLOWER
I20260927 14:59:20.865507 13174 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4818278912b049b787bd28c7aca1c312" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37785 } }
I20260927 14:59:20.865631 13174 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:59:20.865696 13174 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:59:20.865789 13174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [term 1 FOLLOWER]: Advancing to term 2
I20260927 14:59:20.867853 13174 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4818278912b049b787bd28c7aca1c312" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37785 } }
I20260927 14:59:20.868196 13174 leader_election.cc:302] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4818278912b049b787bd28c7aca1c312; no voters:
I20260927 14:59:20.868507 13174 leader_election.cc:288] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 14:59:20.868604 13178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 14:59:20.868803 13178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [term 2 LEADER]: Becoming Leader. State: Replica: 4818278912b049b787bd28c7aca1c312, State: Running, Role: LEADER
I20260927 14:59:20.869125 13178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4818278912b049b787bd28c7aca1c312" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37785 } }
I20260927 14:59:20.869808 13174 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:59:20.870124 13178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4818278912b049b787bd28c7aca1c312. Latest consensus state: current_term: 2 leader_uuid: "4818278912b049b787bd28c7aca1c312" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4818278912b049b787bd28c7aca1c312" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37785 } } }
I20260927 14:59:20.870242 13178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [sys.catalog]: This master's current role is: LEADER
I20260927 14:59:20.870630 13179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4818278912b049b787bd28c7aca1c312" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4818278912b049b787bd28c7aca1c312" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37785 } } }
I20260927 14:59:20.870702 13182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:59:20.871834 13179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312 [sys.catalog]: This master's current role is: LEADER
I20260927 14:59:20.874363 13182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:59:20.874969 13174 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 14:59:20.875180 13182 catalog_manager.cc:1259] Loaded cluster ID: 89aeb2e414164b15af0578d45d300885
I20260927 14:59:20.875281 13182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:59:20.877853 13182 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:59:20.878558 13182 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312: Loaded TSK: 0
I20260927 14:59:20.882217 13182 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 14:59:20.978639 13176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:59:20.979029 13176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:59:20.979100 13176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:59:20.985399 13176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:59:20.985494 13176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:59:20.985582 13176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:59:20.993734 13176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38699/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40761
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37887
--tserver_master_addrs=127.0.98.62:37785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:59:20.995232 13176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:59:20.996605 13176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:59:21.010341 13211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:21.010489 13176 server_base.cc:1034] running on GCE node
W20260927 14:59:21.010761 13214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:21.010341 13212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:21.011559 13176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:59:21.012419 13176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:59:21.015944 13176 hybrid_clock.cc:648] HybridClock initialized: now 1790521161015868 us; error 91 us; skew 500 ppm
Sep 27 14:59:21 dist-test-slave-pz0h krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:59:21.019695 13176 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:59:21.020834 13176 webserver.cc:460] Webserver started at http://127.0.98.1:37887/ using document root <none> and password file <none>
I20260927 14:59:21.021564 13176 fs_manager.cc:359] Metadata directory not provided
I20260927 14:59:21.021709 13176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 14:59:21 dist-test-slave-pz0h krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:59:21.045382 13176 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.000s
I20260927 14:59:21.048887 13222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:21.050617 13176 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260927 14:59:21.050779 13176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "b45bb6d0a0904f2cb868bdcd89d7d5ed"
format_stamp: "Formatted at 2026-09-27 14:59:19 on dist-test-slave-pz0h"
server_key: "9ac4569d1b6d09101ff31a5a5b488d00"
server_key_iv: "7b36e760fdeabf3651f5589e8a1d0a2c"
server_key_version: "kuduclusterkey@0"
I20260927 14:59:21.051566 13176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:59:21.081173 13176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:59:21.085135 13176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:59:21.085325 13176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:59:21.085984 13176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:59:21.086928 13176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:59:21.086979 13176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:21.087035 13176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:59:21.087059 13176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:21.096997 13176 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40761
I20260927 14:59:21.097244 13335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40761 every 8 connection(s)
I20260927 14:59:21.098101 13176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:59:21.101750 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 13176
I20260927 14:59:21.102178 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38699/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46099
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37785
--webserver_port=34463
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38699/kms with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 14:59:21 dist-test-slave-pz0h krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:59:21.129371 13339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:21.100733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37785 (local address 127.0.98.1:54007)
0927 14:59:21.103129 (+ 2396us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:21.103194 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:21.104304 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:21.112191 (+ 7887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:21.112203 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:21.112606 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:59:21.113380 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:21.113406 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:21.116313 (+ 2907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:21.116325 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:21.117953 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:21.117973 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:21.119244 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:21.119250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:21.119480 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:21.120294 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:59:21.120320 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:59:21.122506 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:59:21.125487 (+ 2981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:21.125495 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:21.125514 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:21.125939 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:21.126433 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:21.126437 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:21.126439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:21.126641 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:21.127501 (+ 860us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:59:21.127509 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:59:21.127839 (+ 330us) client_negotiation.cc:769] Sending connection context
0927 14:59:21.128051 (+ 212us) client_negotiation.cc:240] Negotiation successful
0927 14:59:21.128408 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2152,"thread_start_us":111,"threads_started":1}
I20260927 14:59:21.129994 13338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:21.100580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54007 (local address 127.0.98.62:37785)
0927 14:59:21.110147 (+ 9567us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:21.110174 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:21.110238 (+ 64us) server_negotiation.cc:407] Connection header received
0927 14:59:21.110620 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:21.110652 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:21.111222 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:21.113730 (+ 2508us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:21.114081 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:21.116098 (+ 2017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:21.118166 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:21.119016 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:21.121212 (+ 2196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:21.122935 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:21.122944 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:21.122966 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:21.123007 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:21.125304 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:21.126110 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:21.126117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:21.126123 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:21.126293 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:21.126773 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:21.126777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:21.126779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:21.127354 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:21.127489 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:21.128221 (+ 732us) server_negotiation.cc:299] Negotiation successful
0927 14:59:21.128525 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9306,"thread_start_us":118,"threads_started":1}
I20260927 14:59:21.133128 13336 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37785
I20260927 14:59:21.133414 13336 heartbeater.cc:464] Registering TS with master...
I20260927 14:59:21.134055 13336 heartbeater.cc:511] Master 127.0.98.62:37785 requested a full tablet report, sending...
W20260927 14:59:21.136736 13336 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37785 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 14:59:21.136884 13336 heartbeater.cc:464] Registering TS with master...
I20260927 14:59:21.137106 13336 heartbeater.cc:511] Master 127.0.98.62:37785 requested a full tablet report, sending...
I20260927 14:59:21.138353 13336 heartbeater.cc:464] Registering TS with master...
I20260927 14:59:21.138566 13336 heartbeater.cc:511] Master 127.0.98.62:37785 requested a full tablet report, sending...
W20260927 14:59:21.139577 13336 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 14:59:21.243090 13340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:59:21.243366 13340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:59:21.243409 13340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:59:21.247830 13340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:59:21.247905 13340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:59:21.247972 13340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:59:21.253835 13340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38699/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46099
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34463
--tserver_master_addrs=127.0.98.62:37785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:59:21.255046 13340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:59:21.256119 13340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:59:21.266023 13346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:21.266047 13348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:21.266314 13345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:21.266363 13340 server_base.cc:1034] running on GCE node
I20260927 14:59:21.266762 13340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:59:21.267464 13340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:59:21.268646 13340 hybrid_clock.cc:648] HybridClock initialized: now 1790521161268634 us; error 33 us; skew 500 ppm
Sep 27 14:59:21 dist-test-slave-pz0h krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:59:21.271883 13340 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:59:21.273128 13340 webserver.cc:460] Webserver started at http://127.0.98.2:34463/ using document root <none> and password file <none>
I20260927 14:59:21.273881 13340 fs_manager.cc:359] Metadata directory not provided
I20260927 14:59:21.273995 13340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 14:59:21 dist-test-slave-pz0h krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:59:21.298785 13340 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:59:21.301709 13356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:21.303174 13340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:59:21.303308 13340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "9538f91af0ec43fd9c80cdb5d8cf0300"
format_stamp: "Formatted at 2026-09-27 14:59:20 on dist-test-slave-pz0h"
server_key: "a22f87b5ac03211fbd977eb5ae5f6a41"
server_key_iv: "5d8abcb73dc21933e2f522635e870391"
server_key_version: "kuduclusterkey@0"
I20260927 14:59:21.304073 13340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:59:21.326776 13340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:59:21.330358 13340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:59:21.330554 13340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:59:21.331197 13340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:59:21.332080 13340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:59:21.332130 13340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:21.332185 13340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:59:21.332211 13340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:21.340545 13340 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46099
I20260927 14:59:21.341912 13340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:59:21.349689 13469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46099 every 8 connection(s)
I20260927 14:59:21.350167 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 13340
I20260927 14:59:21.350605 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38699/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39971
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37785
--webserver_port=46607
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38699/kms with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 14:59:21 dist-test-slave-pz0h krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:59:21.367794 13338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:21.348684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58779 (local address 127.0.98.62:37785)
0927 14:59:21.348856 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:21.348861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:21.352365 (+ 3504us) server_negotiation.cc:407] Connection header received
0927 14:59:21.354424 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:21.354429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:21.354498 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:21.354559 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:21.356579 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:21.357446 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:21.359462 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:21.359833 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:21.359962 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:21.363136 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:21.363141 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:21.363144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:21.363196 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:21.365031 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:21.365728 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:21.365732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:21.365734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:21.365827 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:21.366239 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:21.366242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:21.366244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:21.366473 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:21.366550 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:21.367449 (+ 899us) server_negotiation.cc:299] Negotiation successful
0927 14:59:21.367601 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 14:59:21.369766 13472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:21.349088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37785 (local address 127.0.98.2:58779)
0927 14:59:21.352127 (+ 3039us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:21.352222 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:21.353438 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:21.355101 (+ 1663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:21.355113 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:21.355531 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:59:21.356350 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:21.356374 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:21.357622 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:21.357626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:21.359288 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:21.359304 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:21.360027 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:21.360032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:21.360281 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:21.361007 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:59:21.361031 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:59:21.362868 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:59:21.365192 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:21.365199 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:21.365220 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:21.365600 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:21.365933 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:21.365935 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:21.365937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:21.366127 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:21.366615 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:59:21.366623 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:59:21.367020 (+ 397us) client_negotiation.cc:769] Sending connection context
0927 14:59:21.368278 (+ 1258us) client_negotiation.cc:240] Negotiation successful
0927 14:59:21.368659 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2755,"thread_start_us":159,"threads_started":1}
I20260927 14:59:21.371806 13470 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37785
I20260927 14:59:21.372119 13470 heartbeater.cc:464] Registering TS with master...
I20260927 14:59:21.372803 13470 heartbeater.cc:511] Master 127.0.98.62:37785 requested a full tablet report, sending...
W20260927 14:59:21.375216 13470 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37785 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 14:59:21.375356 13470 heartbeater.cc:464] Registering TS with master...
I20260927 14:59:21.375556 13470 heartbeater.cc:511] Master 127.0.98.62:37785 requested a full tablet report, sending...
I20260927 14:59:21.376535 13470 heartbeater.cc:464] Registering TS with master...
I20260927 14:59:21.376734 13470 heartbeater.cc:511] Master 127.0.98.62:37785 requested a full tablet report, sending...
W20260927 14:59:21.377660 13470 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 14:59:21.506520 13473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:59:21.506872 13473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:59:21.506947 13473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:59:21.511557 13473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:59:21.511639 13473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:59:21.511726 13473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:59:21.519284 13473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38699/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46607
--tserver_master_addrs=127.0.98.62:37785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:59:21.520785 13473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:59:21.521968 13473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:59:21.533572 13482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:21.533468 13479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:21.533798 13473 server_base.cc:1034] running on GCE node
W20260927 14:59:21.533464 13480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:21.534472 13473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:59:21.535406 13473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:59:21.536636 13473 hybrid_clock.cc:648] HybridClock initialized: now 1790521161536591 us; error 54 us; skew 500 ppm
Sep 27 14:59:21 dist-test-slave-pz0h krb5kdc[12142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:59:21.540604 13473 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:59:21.542539 13473 webserver.cc:460] Webserver started at http://127.0.98.3:46607/ using document root <none> and password file <none>
I20260927 14:59:21.543751 13473 fs_manager.cc:359] Metadata directory not provided
I20260927 14:59:21.543954 13473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 14:59:21 dist-test-slave-pz0h krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:59:21.579126 13473 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260927 14:59:21.587841 13490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:21.589807 13473 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20260927 14:59:21.589980 13473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "a06c38d076704da3bd36b97d24e126df"
format_stamp: "Formatted at 2026-09-27 14:59:20 on dist-test-slave-pz0h"
server_key: "ee6cd30048ad63ecbcaeed6715de823d"
server_key_iv: "1c68a0b3b2d8badee0eebe2ac8a1ca96"
server_key_version: "kuduclusterkey@0"
I20260927 14:59:21.590828 13473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:59:21.629650 13473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:59:21.633556 13473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:59:21.633781 13473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:59:21.634581 13473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:59:21.635776 13473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:59:21.635843 13473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:21.635901 13473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:59:21.635931 13473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:21.647653 13473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39971
I20260927 14:59:21.647689 13603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39971 every 8 connection(s)
I20260927 14:59:21.649130 13473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:59:21.653889 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 13473
Sep 27 14:59:21 dist-test-slave-pz0h krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:59:21.668574 13338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:21.652431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45323 (local address 127.0.98.62:37785)
0927 14:59:21.652727 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:21.652734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:21.654108 (+ 1374us) server_negotiation.cc:407] Connection header received
0927 14:59:21.655633 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:21.655637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:21.655704 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:21.655785 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:21.657494 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:21.658322 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:21.660188 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:21.660535 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:21.660611 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:21.663991 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:21.663996 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:21.663999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:21.664045 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:21.665807 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:21.666570 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:21.666575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:21.666577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:21.666676 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:21.667167 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:21.667171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:21.667173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:21.667408 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:21.667493 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:21.668262 (+ 769us) server_negotiation.cc:299] Negotiation successful
0927 14:59:21.668400 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 14:59:21.670583 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:21.653320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37785 (local address 127.0.98.3:45323)
0927 14:59:21.653909 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:21.653969 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:21.655370 (+ 1401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:21.656087 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:21.656099 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:21.656510 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:59:21.657298 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:21.657321 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:21.658504 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:21.658508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:21.660038 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:21.660051 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:21.660829 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:21.660833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:21.661077 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:21.661797 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:59:21.661820 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:59:21.663724 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:59:21.665979 (+ 2255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:21.665987 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:21.666023 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:21.666435 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:21.666821 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:21.666825 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:21.666827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:21.667047 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:21.667563 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:59:21.667572 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:59:21.667952 (+ 380us) client_negotiation.cc:769] Sending connection context
0927 14:59:21.668987 (+ 1035us) client_negotiation.cc:240] Negotiation successful
0927 14:59:21.669394 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":154,"threads_started":1}
I20260927 14:59:21.671674 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:21.654720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39444 (local address 127.0.98.62:37785)
0927 14:59:21.655851 (+ 1131us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:21.655855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:21.656115 (+ 260us) server_negotiation.cc:407] Connection header received
0927 14:59:21.656234 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:21.656237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:21.656277 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:21.656349 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:21.657182 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:21.657768 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:21.658699 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:21.659046 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:21.659143 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:21.668283 (+ 9140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:21.668287 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:21.668290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:21.668318 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:21.669813 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:21.670304 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:21.670308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:21.670310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:21.670399 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:21.670727 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:21.670730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:21.670732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:21.670964 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:21.671209 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:21.671380 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 14:59:21.671502 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1051,"thread_start_us":107,"threads_started":1}
I20260927 14:59:21.672746 13604 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37785
I20260927 14:59:21.673005 13604 heartbeater.cc:464] Registering TS with master...
I20260927 14:59:21.673620 13604 heartbeater.cc:511] Master 127.0.98.62:37785 requested a full tablet report, sending...
W20260927 14:59:21.676025 13604 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37785 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 14:59:21.676174 13604 heartbeater.cc:464] Registering TS with master...
I20260927 14:59:21.676397 13604 heartbeater.cc:511] Master 127.0.98.62:37785 requested a full tablet report, sending...
I20260927 14:59:21.677345 13604 heartbeater.cc:464] Registering TS with master...
I20260927 14:59:21.677547 13604 heartbeater.cc:511] Master 127.0.98.62:37785 requested a full tablet report, sending...
W20260927 14:59:21.678346 13604 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 14:59:22 dist-test-slave-pz0h krb5kdc[12142](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790521162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:59:22.139822 13336 heartbeater.cc:464] Registering TS with master...
I20260927 14:59:22.140142 13336 heartbeater.cc:511] Master 127.0.98.62:37785 requested a full tablet report, sending...
I20260927 14:59:22.141531 13138 ts_manager.cc:194] Registered new tserver with Master: b45bb6d0a0904f2cb868bdcd89d7d5ed (127.0.98.1:40761)
I20260927 14:59:22.143378 13138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54007
I20260927 14:59:22.377936 13470 heartbeater.cc:464] Registering TS with master...
I20260927 14:59:22.378304 13470 heartbeater.cc:511] Master 127.0.98.62:37785 requested a full tablet report, sending...
I20260927 14:59:22.379386 13138 ts_manager.cc:194] Registered new tserver with Master: 9538f91af0ec43fd9c80cdb5d8cf0300 (127.0.98.2:46099)
I20260927 14:59:22.380263 13138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58779
Sep 27 14:59:22 dist-test-slave-pz0h krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:59:22.678596 13604 heartbeater.cc:464] Registering TS with master...
I20260927 14:59:22.678932 13604 heartbeater.cc:511] Master 127.0.98.62:37785 requested a full tablet report, sending...
I20260927 14:59:22.679693 13138 ts_manager.cc:194] Registered new tserver with Master: a06c38d076704da3bd36b97d24e126df (127.0.98.3:39971)
I20260927 14:59:22.680521 13138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45323
I20260927 14:59:22.691651 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:59:22.703990 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:22.693383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39456 (local address 127.0.98.62:37785)
0927 14:59:22.694146 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:22.694151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:22.694166 (+ 15us) server_negotiation.cc:407] Connection header received
0927 14:59:22.694212 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:22.694216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:22.694266 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:22.694364 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:22.695278 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:22.696348 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:22.697677 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:22.698024 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:22.698549 (+ 525us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:22.699899 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:22.699904 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:22.699907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:22.699942 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:22.701671 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:22.702203 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:22.702206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:22.702207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:22.702297 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:22.702592 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:22.702596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:22.702598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:22.702872 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:22.703601 (+ 729us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:22.703688 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 14:59:22.703818 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":99,"threads_started":1}
I20260927 14:59:22.708601 13138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:59:22.714282 13138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 27 14:59:22 dist-test-slave-pz0h krb5kdc[12142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 14:59:22.748639 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:22.736867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40761 (local address 127.0.0.1:39182)
0927 14:59:22.739201 (+ 2334us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:22.739212 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:22.739281 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:22.741230 (+ 1949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:22.741237 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:22.741268 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:59:22.741439 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:22.741445 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:22.744069 (+ 2624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:22.744072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:22.745463 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:22.745472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:22.748051 (+ 2579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:22.748055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:22.748201 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:22.748251 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 14:59:22.748339 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 14:59:22.748477 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2274,"thread_start_us":122,"threads_started":1}
I20260927 14:59:22.749782 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:22.737043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39182 (local address 127.0.98.1:40761)
0927 14:59:22.739743 (+ 2700us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:22.739750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:22.739775 (+ 25us) server_negotiation.cc:407] Connection header received
0927 14:59:22.739852 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:22.739858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:22.741039 (+ 1181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:22.741608 (+ 569us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:59:22.741792 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:22.743948 (+ 2156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:22.745785 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:22.747910 (+ 2125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:22.749290 (+ 1380us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:22.749470 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:22.749533 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 14:59:22.749642 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2578,"thread_start_us":102,"threads_started":1}
I20260927 14:59:22.753464 13269 tablet_service.cc:1467] Processing CreateTablet for tablet 0bd02f9dc5a6479e93516afdf82c7c50 (DEFAULT_TABLE table=test-table [id=12aecaad8175411c975db6a484705b40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:59:22.754673 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:22.735101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46099 (local address 127.0.0.1:54306)
0927 14:59:22.738747 (+ 3646us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:22.738850 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:22.738985 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:22.743622 (+ 4637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:22.743626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:22.743646 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:59:22.743763 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:22.743771 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:22.745372 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:22.745375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:22.746691 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:22.746701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:22.754195 (+ 7494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:22.754201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:22.754311 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:22.754339 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 14:59:22.754421 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 14:59:22.754510 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3436,"thread_start_us":130,"threads_started":1}
I20260927 14:59:22.756021 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:22.737753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40576 (local address 127.0.98.3:39971)
0927 14:59:22.747066 (+ 9313us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:22.747074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:22.747103 (+ 29us) server_negotiation.cc:407] Connection header received
0927 14:59:22.747190 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:22.747196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:22.747454 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:22.747944 (+ 490us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:59:22.748118 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:22.749126 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:22.750500 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:22.751359 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:22.755463 (+ 4104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:22.755659 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:22.755743 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 14:59:22.755869 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9173,"thread_start_us":63,"threads_started":1}
I20260927 14:59:22.756023 13269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bd02f9dc5a6479e93516afdf82c7c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:59:22.755339 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:22.737589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39971 (local address 127.0.0.1:40576)
0927 14:59:22.739429 (+ 1840us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:22.739440 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:22.739500 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:22.747627 (+ 8127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:22.747630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:22.747652 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:59:22.747770 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:22.747777 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:22.749264 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:22.749267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:22.750340 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:22.750353 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:22.754925 (+ 4572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:22.754929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:22.755030 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:22.755058 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 14:59:22.755113 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 14:59:22.755199 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1791,"thread_start_us":90,"threads_started":1}
I20260927 14:59:22.760673 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:22.735228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54306 (local address 127.0.98.2:46099)
0927 14:59:22.743050 (+ 7822us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:22.743057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:22.743086 (+ 29us) server_negotiation.cc:407] Connection header received
0927 14:59:22.743183 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:22.743189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:22.743434 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:22.743930 (+ 496us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:59:22.744090 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:22.745236 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:22.746878 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:22.747729 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:22.752042 (+ 4313us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:22.752246 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:22.760352 (+ 8106us) server_negotiation.cc:299] Negotiation successful
0927 14:59:22.760513 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7684,"thread_start_us":97,"threads_started":1}
I20260927 14:59:22.762470 13522 tablet_service.cc:1467] Processing CreateTablet for tablet 0bd02f9dc5a6479e93516afdf82c7c50 (DEFAULT_TABLE table=test-table [id=12aecaad8175411c975db6a484705b40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:59:22.763742 13522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bd02f9dc5a6479e93516afdf82c7c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:59:22.764997 13404 tablet_service.cc:1467] Processing CreateTablet for tablet 0bd02f9dc5a6479e93516afdf82c7c50 (DEFAULT_TABLE table=test-table [id=12aecaad8175411c975db6a484705b40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:59:22.766117 13404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bd02f9dc5a6479e93516afdf82c7c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:59:22.779914 13640 tablet_bootstrap.cc:492] T 0bd02f9dc5a6479e93516afdf82c7c50 P a06c38d076704da3bd36b97d24e126df: Bootstrap starting.
I20260927 14:59:22.781107 13639 tablet_bootstrap.cc:492] T 0bd02f9dc5a6479e93516afdf82c7c50 P 9538f91af0ec43fd9c80cdb5d8cf0300: Bootstrap starting.
I20260927 14:59:22.783527 13640 tablet_bootstrap.cc:654] T 0bd02f9dc5a6479e93516afdf82c7c50 P a06c38d076704da3bd36b97d24e126df: Neither blocks nor log segments found. Creating new log.
I20260927 14:59:22.784106 13638 tablet_bootstrap.cc:492] T 0bd02f9dc5a6479e93516afdf82c7c50 P b45bb6d0a0904f2cb868bdcd89d7d5ed: Bootstrap starting.
I20260927 14:59:22.784541 13640 log.cc:824] T 0bd02f9dc5a6479e93516afdf82c7c50 P a06c38d076704da3bd36b97d24e126df: Log is configured to *not* fsync() on all Append() calls
I20260927 14:59:22.786679 13638 tablet_bootstrap.cc:654] T 0bd02f9dc5a6479e93516afdf82c7c50 P b45bb6d0a0904f2cb868bdcd89d7d5ed: Neither blocks nor log segments found. Creating new log.
I20260927 14:59:22.787149 13639 tablet_bootstrap.cc:654] T 0bd02f9dc5a6479e93516afdf82c7c50 P 9538f91af0ec43fd9c80cdb5d8cf0300: Neither blocks nor log segments found. Creating new log.
I20260927 14:59:22.788232 13639 log.cc:824] T 0bd02f9dc5a6479e93516afdf82c7c50 P 9538f91af0ec43fd9c80cdb5d8cf0300: Log is configured to *not* fsync() on all Append() calls
I20260927 14:59:22.789676 13638 log.cc:824] T 0bd02f9dc5a6479e93516afdf82c7c50 P b45bb6d0a0904f2cb868bdcd89d7d5ed: Log is configured to *not* fsync() on all Append() calls
I20260927 14:59:22.792452 13639 tablet_bootstrap.cc:492] T 0bd02f9dc5a6479e93516afdf82c7c50 P 9538f91af0ec43fd9c80cdb5d8cf0300: No bootstrap required, opened a new log
I20260927 14:59:22.792452 13640 tablet_bootstrap.cc:492] T 0bd02f9dc5a6479e93516afdf82c7c50 P a06c38d076704da3bd36b97d24e126df: No bootstrap required, opened a new log
I20260927 14:59:22.792693 13640 ts_tablet_manager.cc:1397] T 0bd02f9dc5a6479e93516afdf82c7c50 P a06c38d076704da3bd36b97d24e126df: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260927 14:59:22.792702 13639 ts_tablet_manager.cc:1397] T 0bd02f9dc5a6479e93516afdf82c7c50 P 9538f91af0ec43fd9c80cdb5d8cf0300: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260927 14:59:22.798482 13639 raft_consensus.cc:348] T 0bd02f9dc5a6479e93516afdf82c7c50 P 9538f91af0ec43fd9c80cdb5d8cf0300 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9538f91af0ec43fd9c80cdb5d8cf0300" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46099 } } peers { permanent_uuid: "b45bb6d0a0904f2cb868bdcd89d7d5ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 } } peers { permanent_uuid: "a06c38d076704da3bd36b97d24e126df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } }
I20260927 14:59:22.800283 13640 raft_consensus.cc:348] T 0bd02f9dc5a6479e93516afdf82c7c50 P a06c38d076704da3bd36b97d24e126df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9538f91af0ec43fd9c80cdb5d8cf0300" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46099 } } peers { permanent_uuid: "b45bb6d0a0904f2cb868bdcd89d7d5ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 } } peers { permanent_uuid: "a06c38d076704da3bd36b97d24e126df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } }
I20260927 14:59:22.800665 13640 raft_consensus.cc:374] T 0bd02f9dc5a6479e93516afdf82c7c50 P a06c38d076704da3bd36b97d24e126df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:59:22.800753 13640 raft_consensus.cc:729] T 0bd02f9dc5a6479e93516afdf82c7c50 P a06c38d076704da3bd36b97d24e126df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a06c38d076704da3bd36b97d24e126df, State: Initialized, Role: FOLLOWER
I20260927 14:59:22.801523 13638 tablet_bootstrap.cc:492] T 0bd02f9dc5a6479e93516afdf82c7c50 P b45bb6d0a0904f2cb868bdcd89d7d5ed: No bootstrap required, opened a new log
I20260927 14:59:22.801852 13638 ts_tablet_manager.cc:1397] T 0bd02f9dc5a6479e93516afdf82c7c50 P b45bb6d0a0904f2cb868bdcd89d7d5ed: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20260927 14:59:22.802573 13640 consensus_queue.cc:260] T 0bd02f9dc5a6479e93516afdf82c7c50 P a06c38d076704da3bd36b97d24e126df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9538f91af0ec43fd9c80cdb5d8cf0300" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46099 } } peers { permanent_uuid: "b45bb6d0a0904f2cb868bdcd89d7d5ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 } } peers { permanent_uuid: "a06c38d076704da3bd36b97d24e126df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } }
I20260927 14:59:22.803884 13640 ts_tablet_manager.cc:1428] T 0bd02f9dc5a6479e93516afdf82c7c50 P a06c38d076704da3bd36b97d24e126df: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260927 14:59:22.804329 13604 heartbeater.cc:503] Master 127.0.98.62:37785 was elected leader, sending a full tablet report...
I20260927 14:59:22.806785 13638 raft_consensus.cc:348] T 0bd02f9dc5a6479e93516afdf82c7c50 P b45bb6d0a0904f2cb868bdcd89d7d5ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9538f91af0ec43fd9c80cdb5d8cf0300" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46099 } } peers { permanent_uuid: "b45bb6d0a0904f2cb868bdcd89d7d5ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 } } peers { permanent_uuid: "a06c38d076704da3bd36b97d24e126df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } }
I20260927 14:59:22.807264 13639 raft_consensus.cc:374] T 0bd02f9dc5a6479e93516afdf82c7c50 P 9538f91af0ec43fd9c80cdb5d8cf0300 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:59:22.807391 13639 raft_consensus.cc:729] T 0bd02f9dc5a6479e93516afdf82c7c50 P 9538f91af0ec43fd9c80cdb5d8cf0300 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9538f91af0ec43fd9c80cdb5d8cf0300, State: Initialized, Role: FOLLOWER
I20260927 14:59:22.808018 13639 consensus_queue.cc:260] T 0bd02f9dc5a6479e93516afdf82c7c50 P 9538f91af0ec43fd9c80cdb5d8cf0300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9538f91af0ec43fd9c80cdb5d8cf0300" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46099 } } peers { permanent_uuid: "b45bb6d0a0904f2cb868bdcd89d7d5ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 } } peers { permanent_uuid: "a06c38d076704da3bd36b97d24e126df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } }
I20260927 14:59:22.808894 13470 heartbeater.cc:503] Master 127.0.98.62:37785 was elected leader, sending a full tablet report...
I20260927 14:59:22.810156 13638 raft_consensus.cc:374] T 0bd02f9dc5a6479e93516afdf82c7c50 P b45bb6d0a0904f2cb868bdcd89d7d5ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:59:22.810280 13638 raft_consensus.cc:729] T 0bd02f9dc5a6479e93516afdf82c7c50 P b45bb6d0a0904f2cb868bdcd89d7d5ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45bb6d0a0904f2cb868bdcd89d7d5ed, State: Initialized, Role: FOLLOWER
I20260927 14:59:22.811064 13639 ts_tablet_manager.cc:1428] T 0bd02f9dc5a6479e93516afdf82c7c50 P 9538f91af0ec43fd9c80cdb5d8cf0300: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260927 14:59:22.811828 13638 consensus_queue.cc:260] T 0bd02f9dc5a6479e93516afdf82c7c50 P b45bb6d0a0904f2cb868bdcd89d7d5ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9538f91af0ec43fd9c80cdb5d8cf0300" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46099 } } peers { permanent_uuid: "b45bb6d0a0904f2cb868bdcd89d7d5ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 } } peers { permanent_uuid: "a06c38d076704da3bd36b97d24e126df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } }
I20260927 14:59:22.812541 13336 heartbeater.cc:503] Master 127.0.98.62:37785 was elected leader, sending a full tablet report...
I20260927 14:59:22.812934 13638 ts_tablet_manager.cc:1428] T 0bd02f9dc5a6479e93516afdf82c7c50 P b45bb6d0a0904f2cb868bdcd89d7d5ed: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
W20260927 14:59:22.850332 13471 tablet.cc:2369] T 0bd02f9dc5a6479e93516afdf82c7c50 P 9538f91af0ec43fd9c80cdb5d8cf0300: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 14:59:22.853094 13337 tablet.cc:2369] T 0bd02f9dc5a6479e93516afdf82c7c50 P b45bb6d0a0904f2cb868bdcd89d7d5ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 14:59:22.901127 13605 tablet.cc:2369] T 0bd02f9dc5a6479e93516afdf82c7c50 P a06c38d076704da3bd36b97d24e126df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:59:23.023855 13647 raft_consensus.cc:482] T 0bd02f9dc5a6479e93516afdf82c7c50 P b45bb6d0a0904f2cb868bdcd89d7d5ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:59:23.024142 13647 raft_consensus.cc:504] T 0bd02f9dc5a6479e93516afdf82c7c50 P b45bb6d0a0904f2cb868bdcd89d7d5ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9538f91af0ec43fd9c80cdb5d8cf0300" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46099 } } peers { permanent_uuid: "b45bb6d0a0904f2cb868bdcd89d7d5ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 } } peers { permanent_uuid: "a06c38d076704da3bd36b97d24e126df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } }
I20260927 14:59:23.025619 13647 leader_election.cc:288] T 0bd02f9dc5a6479e93516afdf82c7c50 P b45bb6d0a0904f2cb868bdcd89d7d5ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9538f91af0ec43fd9c80cdb5d8cf0300 (127.0.98.2:46099), a06c38d076704da3bd36b97d24e126df (127.0.98.3:39971)
I20260927 14:59:23.031240 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:23.026678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39971 (local address 127.0.98.1:35143)
0927 14:59:23.027229 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:23.027247 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:23.027384 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:23.027654 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:23.027657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:23.027678 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:59:23.027861 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:23.027868 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:23.028608 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:23.028612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:23.029852 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:23.029863 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:23.030837 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:23.030840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:23.030927 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:23.030948 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 14:59:23.031011 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 14:59:23.031086 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":4,"thread_start_us":276,"threads_started":1}
I20260927 14:59:23.031219 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:23.026314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41415 (local address 127.0.98.2:46099)
0927 14:59:23.026520 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:23.026526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:23.026692 (+ 166us) server_negotiation.cc:407] Connection header received
0927 14:59:23.027309 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:23.027314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:23.027380 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:23.027461 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:59:23.028018 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:23.028791 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:23.030124 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:23.030541 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:23.030637 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:23.030708 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:23.030952 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 14:59:23.031054 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 14:59:23.031257 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:23.026880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35143 (local address 127.0.98.3:39971)
0927 14:59:23.027031 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:23.027035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:23.027342 (+ 307us) server_negotiation.cc:407] Connection header received
0927 14:59:23.027482 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:23.027485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:23.027547 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:23.027620 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:59:23.027990 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:23.028504 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:23.030006 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:23.030708 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:23.030844 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:23.030914 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:23.031037 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 14:59:23.031118 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 14:59:23.031719 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:23.026138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46099 (local address 127.0.98.1:41415)
0927 14:59:23.026578 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:23.026596 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:23.027079 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:23.027714 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:23.027716 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:23.027743 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:59:23.027881 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:23.027886 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:23.028915 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:23.028919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:23.030017 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:23.030026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:23.030668 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:23.030672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:23.030769 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:23.030830 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 14:59:23.031480 (+ 650us) client_negotiation.cc:240] Negotiation successful
0927 14:59:23.031566 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":19,"thread_start_us":173,"threads_started":1}
I20260927 14:59:23.032536 13558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd02f9dc5a6479e93516afdf82c7c50" candidate_uuid: "b45bb6d0a0904f2cb868bdcd89d7d5ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a06c38d076704da3bd36b97d24e126df" is_pre_election: true
I20260927 14:59:23.032613 13424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd02f9dc5a6479e93516afdf82c7c50" candidate_uuid: "b45bb6d0a0904f2cb868bdcd89d7d5ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9538f91af0ec43fd9c80cdb5d8cf0300" is_pre_election: true
I20260927 14:59:23.032894 13558 raft_consensus.cc:2454] T 0bd02f9dc5a6479e93516afdf82c7c50 P a06c38d076704da3bd36b97d24e126df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b45bb6d0a0904f2cb868bdcd89d7d5ed in term 0.
I20260927 14:59:23.032903 13424 raft_consensus.cc:2454] T 0bd02f9dc5a6479e93516afdf82c7c50 P 9538f91af0ec43fd9c80cdb5d8cf0300 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b45bb6d0a0904f2cb868bdcd89d7d5ed in term 0.
I20260927 14:59:23.033382 13226 leader_election.cc:302] T 0bd02f9dc5a6479e93516afdf82c7c50 P b45bb6d0a0904f2cb868bdcd89d7d5ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a06c38d076704da3bd36b97d24e126df, b45bb6d0a0904f2cb868bdcd89d7d5ed; no voters:
I20260927 14:59:23.033680 13647 raft_consensus.cc:2781] T 0bd02f9dc5a6479e93516afdf82c7c50 P b45bb6d0a0904f2cb868bdcd89d7d5ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:59:23.033792 13647 raft_consensus.cc:482] T 0bd02f9dc5a6479e93516afdf82c7c50 P b45bb6d0a0904f2cb868bdcd89d7d5ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:59:23.033844 13647 raft_consensus.cc:3037] T 0bd02f9dc5a6479e93516afdf82c7c50 P b45bb6d0a0904f2cb868bdcd89d7d5ed [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:59:23.035274 13647 raft_consensus.cc:504] T 0bd02f9dc5a6479e93516afdf82c7c50 P b45bb6d0a0904f2cb868bdcd89d7d5ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9538f91af0ec43fd9c80cdb5d8cf0300" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46099 } } peers { permanent_uuid: "b45bb6d0a0904f2cb868bdcd89d7d5ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 } } peers { permanent_uuid: "a06c38d076704da3bd36b97d24e126df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } }
I20260927 14:59:23.035717 13647 leader_election.cc:288] T 0bd02f9dc5a6479e93516afdf82c7c50 P b45bb6d0a0904f2cb868bdcd89d7d5ed [CANDIDATE]: Term 1 election: Requested vote from peers 9538f91af0ec43fd9c80cdb5d8cf0300 (127.0.98.2:46099), a06c38d076704da3bd36b97d24e126df (127.0.98.3:39971)
I20260927 14:59:23.036257 13424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd02f9dc5a6479e93516afdf82c7c50" candidate_uuid: "b45bb6d0a0904f2cb868bdcd89d7d5ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9538f91af0ec43fd9c80cdb5d8cf0300"
I20260927 14:59:23.036331 13558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd02f9dc5a6479e93516afdf82c7c50" candidate_uuid: "b45bb6d0a0904f2cb868bdcd89d7d5ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a06c38d076704da3bd36b97d24e126df"
I20260927 14:59:23.036399 13424 raft_consensus.cc:3037] T 0bd02f9dc5a6479e93516afdf82c7c50 P 9538f91af0ec43fd9c80cdb5d8cf0300 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:59:23.036471 13558 raft_consensus.cc:3037] T 0bd02f9dc5a6479e93516afdf82c7c50 P a06c38d076704da3bd36b97d24e126df [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:59:23.037503 13424 raft_consensus.cc:2454] T 0bd02f9dc5a6479e93516afdf82c7c50 P 9538f91af0ec43fd9c80cdb5d8cf0300 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b45bb6d0a0904f2cb868bdcd89d7d5ed in term 1.
I20260927 14:59:23.037730 13558 raft_consensus.cc:2454] T 0bd02f9dc5a6479e93516afdf82c7c50 P a06c38d076704da3bd36b97d24e126df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b45bb6d0a0904f2cb868bdcd89d7d5ed in term 1.
I20260927 14:59:23.037927 13226 leader_election.cc:302] T 0bd02f9dc5a6479e93516afdf82c7c50 P b45bb6d0a0904f2cb868bdcd89d7d5ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9538f91af0ec43fd9c80cdb5d8cf0300, b45bb6d0a0904f2cb868bdcd89d7d5ed; no voters:
I20260927 14:59:23.038111 13647 raft_consensus.cc:2781] T 0bd02f9dc5a6479e93516afdf82c7c50 P b45bb6d0a0904f2cb868bdcd89d7d5ed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:59:23.038336 13647 raft_consensus.cc:686] T 0bd02f9dc5a6479e93516afdf82c7c50 P b45bb6d0a0904f2cb868bdcd89d7d5ed [term 1 LEADER]: Becoming Leader. State: Replica: b45bb6d0a0904f2cb868bdcd89d7d5ed, State: Running, Role: LEADER
I20260927 14:59:23.038731 13647 consensus_queue.cc:237] T 0bd02f9dc5a6479e93516afdf82c7c50 P b45bb6d0a0904f2cb868bdcd89d7d5ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9538f91af0ec43fd9c80cdb5d8cf0300" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46099 } } peers { permanent_uuid: "b45bb6d0a0904f2cb868bdcd89d7d5ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 } } peers { permanent_uuid: "a06c38d076704da3bd36b97d24e126df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } }
I20260927 14:59:23.042274 13138 catalog_manager.cc:5495] T 0bd02f9dc5a6479e93516afdf82c7c50 P b45bb6d0a0904f2cb868bdcd89d7d5ed reported cstate change: term changed from 0 to 1, leader changed from <none> to b45bb6d0a0904f2cb868bdcd89d7d5ed (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b45bb6d0a0904f2cb868bdcd89d7d5ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9538f91af0ec43fd9c80cdb5d8cf0300" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b45bb6d0a0904f2cb868bdcd89d7d5ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a06c38d076704da3bd36b97d24e126df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 } health_report { overall_health: UNKNOWN } } }
I20260927 14:59:23.112109 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:23.108515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39198 (local address 127.0.98.1:40761)
0927 14:59:23.108661 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:23.108665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:23.108678 (+ 13us) server_negotiation.cc:407] Connection header received
0927 14:59:23.108755 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:23.108758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:23.108814 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:23.108901 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:59:23.109216 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:23.109766 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:23.110640 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:23.110939 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:23.111024 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:23.111292 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:59:23.111739 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:59:23.111825 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:23.111888 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 14:59:23.111945 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 14:59:23.118083 13558 raft_consensus.cc:1261] T 0bd02f9dc5a6479e93516afdf82c7c50 P a06c38d076704da3bd36b97d24e126df [term 1 FOLLOWER]: Refusing update from remote peer b45bb6d0a0904f2cb868bdcd89d7d5ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:59:23.118083 13424 raft_consensus.cc:1261] T 0bd02f9dc5a6479e93516afdf82c7c50 P 9538f91af0ec43fd9c80cdb5d8cf0300 [term 1 FOLLOWER]: Refusing update from remote peer b45bb6d0a0904f2cb868bdcd89d7d5ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:59:23.118834 13647 consensus_queue.cc:1035] T 0bd02f9dc5a6479e93516afdf82c7c50 P b45bb6d0a0904f2cb868bdcd89d7d5ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a06c38d076704da3bd36b97d24e126df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:59:23.119041 13657 consensus_queue.cc:1035] T 0bd02f9dc5a6479e93516afdf82c7c50 P b45bb6d0a0904f2cb868bdcd89d7d5ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "9538f91af0ec43fd9c80cdb5d8cf0300" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:59:23.127763 13661 mvcc.cc:205] Tried to move back new op lower bound from 7333974684124291072 to 7333974683809464320. Current Snapshot: MvccSnapshot[applied={T|T < 7333974684124291072}]
I20260927 14:59:23.130753 13662 mvcc.cc:205] Tried to move back new op lower bound from 7333974684124291072 to 7333974683809464320. Current Snapshot: MvccSnapshot[applied={T|T < 7333974684124291072 or (T in {7333974684124291072})}]
I20260927 14:59:23.136552 13138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:59:23.136736 13138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:59:23.139129 13138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4818278912b049b787bd28c7aca1c312: Sending DeleteTablet for 3 replicas of tablet 0bd02f9dc5a6479e93516afdf82c7c50
I20260927 14:59:23.139966 13404 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bd02f9dc5a6479e93516afdf82c7c50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:59:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54306
I20260927 14:59:23.140038 13269 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bd02f9dc5a6479e93516afdf82c7c50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:59:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39182
I20260927 14:59:23.140244 13522 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bd02f9dc5a6479e93516afdf82c7c50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:59:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40576
I20260927 14:59:23.140512 13670 tablet_replica.cc:330] T 0bd02f9dc5a6479e93516afdf82c7c50 P b45bb6d0a0904f2cb868bdcd89d7d5ed: stopping tablet replica
I20260927 14:59:23.140818 13670 raft_consensus.cc:2229] T 0bd02f9dc5a6479e93516afdf82c7c50 P b45bb6d0a0904f2cb868bdcd89d7d5ed [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:59:23.140937 13669 tablet_replica.cc:330] T 0bd02f9dc5a6479e93516afdf82c7c50 P 9538f91af0ec43fd9c80cdb5d8cf0300: stopping tablet replica
I20260927 14:59:23.141057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 13176
I20260927 14:59:23.141811 13671 tablet_replica.cc:330] T 0bd02f9dc5a6479e93516afdf82c7c50 P a06c38d076704da3bd36b97d24e126df: stopping tablet replica
I20260927 14:59:23.142113 13671 raft_consensus.cc:2229] T 0bd02f9dc5a6479e93516afdf82c7c50 P a06c38d076704da3bd36b97d24e126df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:59:23.142426 13671 raft_consensus.cc:2258] T 0bd02f9dc5a6479e93516afdf82c7c50 P a06c38d076704da3bd36b97d24e126df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:59:23.143894 13671 ts_tablet_manager.cc:1905] T 0bd02f9dc5a6479e93516afdf82c7c50 P a06c38d076704da3bd36b97d24e126df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:59:23.146170 13671 ts_tablet_manager.cc:1918] T 0bd02f9dc5a6479e93516afdf82c7c50 P a06c38d076704da3bd36b97d24e126df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:59:23.146270 13671 log.cc:1196] T 0bd02f9dc5a6479e93516afdf82c7c50 P a06c38d076704da3bd36b97d24e126df: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790521004237705-392-0/minicluster-data/ts-2/wal/wals/0bd02f9dc5a6479e93516afdf82c7c50
I20260927 14:59:23.146549 13671 ts_tablet_manager.cc:1939] T 0bd02f9dc5a6479e93516afdf82c7c50 P a06c38d076704da3bd36b97d24e126df: Deleting consensus metadata
W20260927 14:59:23.147265 13125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40761 (error 108)
I20260927 14:59:23.148119 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 13340
I20260927 14:59:23.148159 13669 raft_consensus.cc:2229] T 0bd02f9dc5a6479e93516afdf82c7c50 P 9538f91af0ec43fd9c80cdb5d8cf0300 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:59:23.148182 13125 catalog_manager.cc:4841] TS a06c38d076704da3bd36b97d24e126df (127.0.98.3:39971): tablet 0bd02f9dc5a6479e93516afdf82c7c50 (table test-table [id=12aecaad8175411c975db6a484705b40]) successfully deleted
I20260927 14:59:23.148311 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:23.148013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40761 (local address 127.0.0.1:39202)
0927 14:59:23.148153 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:23.148225 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 14:59:23.149158 13125 catalog_manager.cc:4584] TS b45bb6d0a0904f2cb868bdcd89d7d5ed (127.0.98.1:40761): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0bd02f9dc5a6479e93516afdf82c7c50: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40761: connect: Connection refused (error 111)
I20260927 14:59:23.153167 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 13473
I20260927 14:59:23.154217 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:23.153257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46099 (local address 127.0.0.1:54316)
0927 14:59:23.154072 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:23.154127 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":733}
I20260927 14:59:23.158093 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 13105
I20260927 14:59:23.163913 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 14:59:23.499687 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 14:59:23.499809 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 14:59:23.835309 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 14:59:23.835 UTC [12172] LOG: received smart shutdown request
2026-09-27 14:59:23.839 UTC [12172] LOG: background worker "logical replication launcher" (PID 12181) exited with exit code 1
2026-09-27 14:59:23.839 UTC [12176] LOG: shutting down
2026-09-27 14:59:24.000 UTC [12172] LOG: database system is shut down
2026-09-27T14:59:24Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50649 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:59:24 dist-test-slave-pz0h krb5kdc[13681](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:59:24 dist-test-slave-pz0h krb5kdc[13681](info): set up 2 sockets
Sep 27 14:59:24 dist-test-slave-pz0h krb5kdc[13681](info): commencing operation
krb5kdc: starting...
W20260927 14:59:25.062637 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 14:59:25 dist-test-slave-pz0h krb5kdc[13681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:59:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:59:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:59:25.180401 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42131
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42131,127.0.98.61:41181,127.0.98.60:34417
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:59:25.294560 13694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:59:25.294955 13694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:59:25.295020 13694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:59:25.299263 13694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:59:25.299345 13694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:59:25.299371 13694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:59:25.299393 13694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:59:25.299413 13694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:59:25.305191 13694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42131,127.0.98.61:41181,127.0.98.60:34417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:59:25.306330 13694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:59:25.307410 13694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:59:25.316044 13699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:25.316238 13694 server_base.cc:1034] running on GCE node
W20260927 14:59:25.316044 13700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:25.316052 13702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:25.317013 13694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:59:25.318239 13694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:59:25.319403 13694 hybrid_clock.cc:648] HybridClock initialized: now 1790521165319384 us; error 40 us; skew 500 ppm
Sep 27 14:59:25 dist-test-slave-pz0h krb5kdc[13681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:59:25.322130 13694 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:59:25.323213 13694 webserver.cc:460] Webserver started at http://127.0.98.62:41207/ using document root <none> and password file <none>
I20260927 14:59:25.323824 13694 fs_manager.cc:359] Metadata directory not provided
I20260927 14:59:25.323896 13694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:59:25.324118 13694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:59:25.326290 13694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "d415904c59af4d8eb27cb8d67307ef74"
format_stamp: "Formatted at 2026-09-27 14:59:25 on dist-test-slave-pz0h"
server_key: "6225066b9b304e6e529bc117cb23e382"
server_key_iv: "78a621f5f29bd68283fca91e03bb1b92"
server_key_version: "encryptionkey@0"
I20260927 14:59:25.326936 13694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "d415904c59af4d8eb27cb8d67307ef74"
format_stamp: "Formatted at 2026-09-27 14:59:25 on dist-test-slave-pz0h"
server_key: "6225066b9b304e6e529bc117cb23e382"
server_key_iv: "78a621f5f29bd68283fca91e03bb1b92"
server_key_version: "encryptionkey@0"
I20260927 14:59:25.330945 13694 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 14:59:25.333789 13709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:25.335337 13694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:59:25.335493 13694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "d415904c59af4d8eb27cb8d67307ef74"
format_stamp: "Formatted at 2026-09-27 14:59:25 on dist-test-slave-pz0h"
server_key: "6225066b9b304e6e529bc117cb23e382"
server_key_iv: "78a621f5f29bd68283fca91e03bb1b92"
server_key_version: "encryptionkey@0"
I20260927 14:59:25.335618 13694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:59:25.353709 13694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:59:25.357000 13694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:59:25.357201 13694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:59:25.364384 13694 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42131
I20260927 14:59:25.364439 13761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42131 every 8 connection(s)
I20260927 14:59:25.365411 13694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:59:25.367130 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 13694
I20260927 14:59:25.367276 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 14:59:25.367569 392 external_mini_cluster.cc:1428] Setting key 480f2c41b11a644478b1eb3de109c9a8
I20260927 14:59:25.368851 13762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:59:25.373076 13762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42131 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 14:59:25 dist-test-slave-pz0h krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:59:25.389637 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.375546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56686 (local address 127.0.98.62:42131)
0927 14:59:25.376074 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:25.376093 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:25.376251 (+ 158us) server_negotiation.cc:407] Connection header received
0927 14:59:25.377503 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:25.377512 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:25.377806 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:25.378015 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:25.379117 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.380382 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.381533 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.382174 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.382370 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:25.384875 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:25.384882 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:25.384899 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:25.384927 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:25.386381 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:25.386943 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:25.386950 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:25.386952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:25.387114 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:25.387458 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:25.387461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:25.387462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:25.387923 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:25.388036 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:25.388414 (+ 378us) server_negotiation.cc:299] Negotiation successful
0927 14:59:25.388788 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":160,"server-negotiator.queue_time_us":397,"spinlock_wait_cycles":48512,"thread_start_us":147,"threads_started":1}
I20260927 14:59:25.389649 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.375546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42131 (local address 127.0.0.1:56686)
0927 14:59:25.376086 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:25.376144 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:25.377119 (+ 975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:25.378039 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:25.378046 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:25.378463 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:59:25.378904 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:25.378925 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.380535 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.380538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:25.381405 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:25.381414 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.382371 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.382375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:25.382484 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:25.383142 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:59:25.383160 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:59:25.384690 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:59:25.386513 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:25.386523 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:25.386534 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:25.386782 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:25.387238 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:25.387241 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:25.387242 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:25.387367 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:25.388036 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:59:25.388042 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:59:25.388188 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 14:59:25.388341 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 14:59:25.388726 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":1,"thread_start_us":144,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 14:59:25.392356 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41181
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42131,127.0.98.61:41181,127.0.98.60:34417
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:59:25.392743 13762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:59:25.393607 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.393278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41181 (local address 127.0.0.1:52392)
0927 14:59:25.393412 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:25.393505 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 14:59:25.393867 13712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41181: connect: Connection refused (error 111)
I20260927 14:59:25.394951 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.394676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41181 (local address 127.0.0.1:52398)
0927 14:59:25.394824 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:25.394879 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 14:59:25.395119 13762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41181: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41181: connect: Connection refused (error 111)
I20260927 14:59:25.444451 13762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } attempt: 1
I20260927 14:59:25.445804 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.445235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41181 (local address 127.0.0.1:52404)
0927 14:59:25.445527 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:25.445672 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260927 14:59:25.446923 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.446564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41181 (local address 127.0.0.1:52408)
0927 14:59:25.446714 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:25.446830 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 14:59:25.447220 13762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41181: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41181: connect: Connection refused (error 111)
W20260927 14:59:25.513432 13768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:59:25.513741 13768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:59:25.513784 13768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 14:59:25.515439 13762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } attempt: 2
I20260927 14:59:25.516386 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.516052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41181 (local address 127.0.0.1:52412)
0927 14:59:25.516231 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:25.516297 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 14:59:25.517081 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.516877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41181 (local address 127.0.0.1:52422)
0927 14:59:25.516975 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:25.517022 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 14:59:25.517190 13762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41181: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41181: connect: Connection refused (error 111)
W20260927 14:59:25.518395 13768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:59:25.518455 13768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:59:25.518473 13768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:59:25.518489 13768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:59:25.518512 13768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:59:25.524255 13768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42131,127.0.98.61:41181,127.0.98.60:34417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:59:25.525352 13768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:59:25.526338 13768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:59:25.535147 13774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:25.535308 13768 server_base.cc:1034] running on GCE node
W20260927 14:59:25.535147 13775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:25.535147 13777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:25.536054 13768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:59:25.537212 13768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:59:25.538388 13768 hybrid_clock.cc:648] HybridClock initialized: now 1790521165538368 us; error 38 us; skew 500 ppm
Sep 27 14:59:25 dist-test-slave-pz0h krb5kdc[13681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:59:25.541706 13768 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 14:59:25.542716 13768 webserver.cc:460] Webserver started at http://127.0.98.61:45601/ using document root <none> and password file <none>
I20260927 14:59:25.543322 13768 fs_manager.cc:359] Metadata directory not provided
I20260927 14:59:25.543392 13768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:59:25.543591 13768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:59:25.545735 13768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-1/data/instance:
uuid: "30f235a41d0d4533a052bfd6c144e453"
format_stamp: "Formatted at 2026-09-27 14:59:25 on dist-test-slave-pz0h"
server_key: "8cc6e4ee69f301d92ef2f95566a250d1"
server_key_iv: "830a732b864963e65ee978508398073c"
server_key_version: "encryptionkey@0"
I20260927 14:59:25.546298 13768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-1/wal/instance:
uuid: "30f235a41d0d4533a052bfd6c144e453"
format_stamp: "Formatted at 2026-09-27 14:59:25 on dist-test-slave-pz0h"
server_key: "8cc6e4ee69f301d92ef2f95566a250d1"
server_key_iv: "830a732b864963e65ee978508398073c"
server_key_version: "encryptionkey@0"
I20260927 14:59:25.550230 13768 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 14:59:25.552819 13784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:25.554148 13768 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:59:25.554266 13768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-1/wal
uuid: "30f235a41d0d4533a052bfd6c144e453"
format_stamp: "Formatted at 2026-09-27 14:59:25 on dist-test-slave-pz0h"
server_key: "8cc6e4ee69f301d92ef2f95566a250d1"
server_key_iv: "830a732b864963e65ee978508398073c"
server_key_version: "encryptionkey@0"
I20260927 14:59:25.554389 13768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:59:25.581280 13768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:59:25.584556 13768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:59:25.584748 13768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:59:25.592059 13768 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41181
I20260927 14:59:25.592101 13836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41181 every 8 connection(s)
I20260927 14:59:25.593047 13768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-1/data/info.pb
I20260927 14:59:25.595944 13837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:59:25.599915 13837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42131 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:59:25.600185 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 13768
I20260927 14:59:25.600294 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-1/wal/instance
I20260927 14:59:25.600533 392 external_mini_cluster.cc:1428] Setting key a6eccec443d92bf304d8d37f4c887afb
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 14:59:25.608355 13762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 14:59:25 dist-test-slave-pz0h krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 14:59:25 dist-test-slave-pz0h krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 14:59:25.616225 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.602036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56700 (local address 127.0.98.62:42131)
0927 14:59:25.602225 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:25.602230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:25.603203 (+ 973us) server_negotiation.cc:407] Connection header received
0927 14:59:25.604302 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:25.604307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:25.604368 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:25.604479 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:25.605820 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.606647 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.607950 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.608189 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.608288 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:25.611192 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:25.611196 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:25.611199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:25.611228 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:25.612956 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:25.613538 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:25.613542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:25.613544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:25.613626 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:25.613963 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:25.613966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:25.613968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:25.614152 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:25.614237 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:25.615936 (+ 1699us) server_negotiation.cc:299] Negotiation successful
0927 14:59:25.616075 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 14:59:25.616314 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.602463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42131 (local address 127.0.0.1:56700)
0927 14:59:25.603019 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:25.603074 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:25.604045 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:25.604696 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:25.604704 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:25.605098 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:59:25.605636 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:25.605651 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.606853 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.606857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:25.607795 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:25.607809 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.608354 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.608359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:25.608725 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:25.609383 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:59:25.609404 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:59:25.611036 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:59:25.613070 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:25.613076 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:25.613096 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:25.613418 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:25.613710 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:25.613713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:25.613715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:25.613864 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:25.614253 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:59:25.614260 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:59:25.614574 (+ 314us) client_negotiation.cc:769] Sending connection context
0927 14:59:25.614748 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 14:59:25.615171 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":151,"threads_started":1}
I20260927 14:59:25.617869 13837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:59:25.618671 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.608804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41181 (local address 127.0.0.1:52432)
0927 14:59:25.608962 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:25.608976 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:25.609076 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:25.609814 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:25.609817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:25.610007 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:59:25.610139 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:25.610147 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.611315 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.611318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:25.612078 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:25.612090 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.612725 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.612729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:25.612828 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:25.613274 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:59:25.613289 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:59:25.614670 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:59:25.616824 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:25.616827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:25.616830 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:25.617088 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:25.617424 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:25.617427 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:25.617429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:25.617535 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:25.618151 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:59:25.618154 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:59:25.618257 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 14:59:25.618362 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 14:59:25.618504 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 14:59:25.619365 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.608927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52432 (local address 127.0.98.61:41181)
0927 14:59:25.609321 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:25.609333 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:25.609359 (+ 26us) server_negotiation.cc:407] Connection header received
0927 14:59:25.609430 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:25.609435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:25.609653 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:25.609811 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:25.610272 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.611184 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.612200 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.612605 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.612705 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:25.614850 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:25.614860 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:25.614875 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:25.614900 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:25.616680 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:25.617219 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:25.617226 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:25.617228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:25.617329 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:25.617629 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:25.617632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:25.617634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:25.617953 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:25.618100 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:25.619004 (+ 904us) server_negotiation.cc:299] Negotiation successful
0927 14:59:25.619172 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":162,"threads_started":1}
I20260927 14:59:25.621260 13762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34417 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:59:25.622016 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.621766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34417 (local address 127.0.0.1:51494)
0927 14:59:25.621889 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:25.621943 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 14:59:25.622869 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.622586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34417 (local address 127.0.0.1:51496)
0927 14:59:25.622730 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:25.622775 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260927 14:59:25.623011 13762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34417: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34417: connect: Connection refused (error 111)
Sep 27 14:59:25 dist-test-slave-pz0h krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 14:59:25.627674 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34417
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42131,127.0.98.61:41181,127.0.98.60:34417
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:59:25.627954 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.618370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41181 (local address 127.0.0.1:52446)
0927 14:59:25.618536 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:25.618550 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:25.618620 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:25.620526 (+ 1906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:25.620529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:25.620699 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:59:25.620817 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:25.620823 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.621696 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.621699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:25.622320 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:25.622331 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.622896 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.622900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:25.623002 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:25.623453 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:59:25.623467 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:59:25.624809 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:59:25.626439 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:25.626442 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:25.626443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:25.626684 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:25.626978 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:25.626981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:25.626982 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:25.627059 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:25.627420 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:59:25.627423 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:59:25.627523 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 14:59:25.627642 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 14:59:25.627797 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 14:59:25.628079 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.618396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52446 (local address 127.0.98.61:41181)
0927 14:59:25.620343 (+ 1947us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:25.620347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:25.620358 (+ 11us) server_negotiation.cc:407] Connection header received
0927 14:59:25.620386 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:25.620388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:25.620422 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:25.620485 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:25.620911 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.621584 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.622435 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.622761 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.622894 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:25.624957 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:25.624959 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:25.624961 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:25.624985 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:25.626329 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:25.626776 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:25.626779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:25.626781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:25.626898 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:25.627146 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:25.627148 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:25.627149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:25.627328 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:25.627396 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:25.627695 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 14:59:25.627883 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":1888,"thread_start_us":117,"threads_started":1}
I20260927 14:59:25.628830 13837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34417 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:59:25.629594 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.629275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34417 (local address 127.0.0.1:51498)
0927 14:59:25.629411 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:25.629513 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 14:59:25.630148 13788 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34417: connect: Connection refused (error 111)
I20260927 14:59:25.631080 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.630840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34417 (local address 127.0.0.1:51500)
0927 14:59:25.630951 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:25.631005 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 14:59:25.631240 13837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34417: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34417: connect: Connection refused (error 111)
I20260927 14:59:25.654199 13762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34417 } attempt: 1
I20260927 14:59:25.655144 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.654765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34417 (local address 127.0.0.1:51512)
0927 14:59:25.654985 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:25.655064 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260927 14:59:25.655819 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.655610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34417 (local address 127.0.0.1:51524)
0927 14:59:25.655712 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:25.655756 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 14:59:25.655936 13762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34417: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34417: connect: Connection refused (error 111)
I20260927 14:59:25.680510 13837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34417 } attempt: 1
I20260927 14:59:25.681433 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.681130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34417 (local address 127.0.0.1:51530)
0927 14:59:25.681288 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:25.681352 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 14:59:25.682116 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.681913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34417 (local address 127.0.0.1:51546)
0927 14:59:25.682012 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:25.682059 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 14:59:25.682296 13837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34417: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34417: connect: Connection refused (error 111)
I20260927 14:59:25.731084 13762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34417 } attempt: 2
I20260927 14:59:25.731884 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.731611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34417 (local address 127.0.0.1:51560)
0927 14:59:25.731748 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:25.731809 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 14:59:25.732613 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.732426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34417 (local address 127.0.0.1:51568)
0927 14:59:25.732513 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:25.732556 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 14:59:25.732731 13762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34417: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34417: connect: Connection refused (error 111)
W20260927 14:59:25.744191 13844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:59:25.744484 13844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:59:25.744526 13844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:59:25.748710 13844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:59:25.748791 13844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:59:25.748809 13844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:59:25.748826 13844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:59:25.748840 13844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:59:25.750478 13837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34417 } attempt: 2
I20260927 14:59:25.751345 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.751066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34417 (local address 127.0.0.1:51570)
0927 14:59:25.751201 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:25.751266 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 14:59:25.752127 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.751881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34417 (local address 127.0.0.1:51580)
0927 14:59:25.752010 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:25.752065 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 14:59:25.752269 13837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34417: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34417: connect: Connection refused (error 111)
I20260927 14:59:25.755093 13844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42131,127.0.98.61:41181,127.0.98.60:34417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:59:25.756294 13844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:59:25.757385 13844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:59:25.766273 13850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:25.766273 13851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:25.766327 13844 server_base.cc:1034] running on GCE node
W20260927 14:59:25.766278 13853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:25.767102 13844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:59:25.768354 13844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:59:25.769559 13844 hybrid_clock.cc:648] HybridClock initialized: now 1790521165769501 us; error 76 us; skew 500 ppm
Sep 27 14:59:25 dist-test-slave-pz0h krb5kdc[13681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:59:25.772620 13844 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 14:59:25.773618 13844 webserver.cc:460] Webserver started at http://127.0.98.60:32835/ using document root <none> and password file <none>
I20260927 14:59:25.774228 13844 fs_manager.cc:359] Metadata directory not provided
I20260927 14:59:25.774295 13844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:59:25.774471 13844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:59:25.776746 13844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-2/data/instance:
uuid: "51780b49284348a7a7ccf9d6ae034c72"
format_stamp: "Formatted at 2026-09-27 14:59:25 on dist-test-slave-pz0h"
server_key: "77e2c90f9185aa110094cd97d18083ab"
server_key_iv: "08eae17dfc599d1c40e82e3b72e23eb9"
server_key_version: "encryptionkey@0"
I20260927 14:59:25.777323 13844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-2/wal/instance:
uuid: "51780b49284348a7a7ccf9d6ae034c72"
format_stamp: "Formatted at 2026-09-27 14:59:25 on dist-test-slave-pz0h"
server_key: "77e2c90f9185aa110094cd97d18083ab"
server_key_iv: "08eae17dfc599d1c40e82e3b72e23eb9"
server_key_version: "encryptionkey@0"
I20260927 14:59:25.781333 13844 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 14:59:25.784096 13860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:25.785527 13844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 14:59:25.785672 13844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-2/wal
uuid: "51780b49284348a7a7ccf9d6ae034c72"
format_stamp: "Formatted at 2026-09-27 14:59:25 on dist-test-slave-pz0h"
server_key: "77e2c90f9185aa110094cd97d18083ab"
server_key_iv: "08eae17dfc599d1c40e82e3b72e23eb9"
server_key_version: "encryptionkey@0"
I20260927 14:59:25.785831 13844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:59:25.813084 13844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:59:25.816556 13844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:59:25.816743 13844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:59:25.823712 13844 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34417
I20260927 14:59:25.823722 13912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34417 every 8 connection(s)
I20260927 14:59:25.824787 13844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-2/data/info.pb
I20260927 14:59:25.827865 13913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:59:25.831909 13762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34417 } attempt: 3
I20260927 14:59:25.832020 13913 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42131 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:59:25.834947 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 13844
I20260927 14:59:25.835073 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790521004237705-392-0/minicluster-data/master-2/wal/instance
I20260927 14:59:25.835389 392 external_mini_cluster.cc:1428] Setting key 5dc8e325bbaf803b2abee7bdfbaaa981
Sep 27 14:59:25 dist-test-slave-pz0h krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 14:59:25 dist-test-slave-pz0h krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 14:59:25 dist-test-slave-pz0h krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 14:59:25.843432 13837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34417 } attempt: 3
I20260927 14:59:25.845890 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.832491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34417 (local address 127.0.0.1:51590)
0927 14:59:25.832645 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:25.832664 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:25.832774 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:25.835123 (+ 2349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:25.835126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:25.835307 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:59:25.835402 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:25.835408 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.837618 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.837622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:25.838406 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:25.838416 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.839075 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.839078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:25.839181 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:25.839743 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:59:25.839754 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:59:25.841291 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:59:25.843703 (+ 2412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:25.843706 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:25.843709 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:25.843955 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:25.844391 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:25.844395 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:25.844397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:25.844489 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:25.845360 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:59:25.845363 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:59:25.845464 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 14:59:25.845558 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 14:59:25.845710 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 14:59:25.846108 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.834024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56716 (local address 127.0.98.62:42131)
0927 14:59:25.834160 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:25.834164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:25.834484 (+ 320us) server_negotiation.cc:407] Connection header received
0927 14:59:25.834953 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:25.834956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:25.835011 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:25.835093 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:25.836504 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.837122 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.838480 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.838844 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.838945 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:25.842073 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:25.842078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:25.842080 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:25.842109 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:25.843660 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:25.844589 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:25.844593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:25.844595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:25.844686 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:25.844987 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:25.844990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:25.844992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:25.845236 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:25.845311 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:25.845852 (+ 541us) server_negotiation.cc:299] Negotiation successful
0927 14:59:25.845971 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 14:59:25.848052 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.833957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42131 (local address 127.0.0.1:56716)
0927 14:59:25.834332 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:25.834389 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:25.834720 (+ 331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:25.835282 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:25.835293 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:25.835756 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:59:25.836369 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:25.836386 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.837231 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.837234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:25.838348 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:25.838359 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.838995 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.839000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:25.839167 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:25.839919 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:59:25.839940 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:59:25.841924 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:59:25.843787 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:25.843798 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:25.843811 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:25.844466 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:25.844796 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:25.844799 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:25.844801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:25.844887 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:25.845393 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:59:25.845401 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:59:25.845582 (+ 181us) client_negotiation.cc:769] Sending connection context
0927 14:59:25.846319 (+ 737us) client_negotiation.cc:240] Negotiation successful
0927 14:59:25.846898 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":253,"thread_start_us":122,"threads_started":1}
I20260927 14:59:25.848069 13915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.833247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51590 (local address 127.0.98.60:34417)
0927 14:59:25.833721 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:25.833738 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:25.833772 (+ 34us) server_negotiation.cc:407] Connection header received
0927 14:59:25.834622 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:25.834647 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:25.834934 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:25.835569 (+ 635us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:25.836029 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.837489 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.838521 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.838948 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.839444 (+ 496us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:25.841434 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:25.841441 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:25.841458 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:25.841485 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:25.843561 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:25.844090 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:25.844097 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:25.844099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:25.844270 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:25.844599 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:25.844603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:25.844605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:25.845248 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:25.846715 (+ 1467us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:25.846775 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 14:59:25.846959 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":322,"thread_start_us":144,"threads_started":1}
I20260927 14:59:25.848069 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.836019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51604 (local address 127.0.98.60:34417)
0927 14:59:25.836386 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:25.836396 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:25.836516 (+ 120us) server_negotiation.cc:407] Connection header received
0927 14:59:25.836631 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:25.836634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:25.836690 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:25.836773 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:25.837638 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.838119 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.839139 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.839418 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.839507 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:25.842430 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:25.842434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:25.842437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:25.842464 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:25.844749 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:25.845256 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:25.845259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:25.845260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:25.845341 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:25.845617 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:25.845621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:25.845622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:25.845837 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:25.846036 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:25.846714 (+ 678us) server_negotiation.cc:299] Negotiation successful
0927 14:59:25.847039 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":142,"server-negotiator.queue_time_us":303,"thread_start_us":105,"threads_started":1}
I20260927 14:59:25.849531 13913 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 14:59:25 dist-test-slave-pz0h krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 14:59:25.853508 13762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74: Bootstrap starting.
I20260927 14:59:25.854000 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.844245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34417 (local address 127.0.0.1:51618)
0927 14:59:25.844450 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:25.844466 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:25.844555 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:25.845029 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:25.845033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:25.845213 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:59:25.845335 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:25.845345 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.846386 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.846390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:25.847183 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:25.847197 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.847914 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.847918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:25.848007 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:25.848557 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:59:25.848573 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:59:25.850106 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:59:25.852320 (+ 2214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:25.852324 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:25.852326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:25.852609 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:25.852948 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:25.852951 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:25.852953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:25.853050 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:25.853487 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:59:25.853490 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:59:25.853603 (+ 113us) client_negotiation.cc:769] Sending connection context
0927 14:59:25.853681 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 14:59:25.853828 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260927 14:59:25.854686 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.844048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51618 (local address 127.0.98.60:34417)
0927 14:59:25.844780 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:25.844788 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:25.844800 (+ 12us) server_negotiation.cc:407] Connection header received
0927 14:59:25.844827 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:25.844830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:25.844871 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:25.845438 (+ 567us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:25.845521 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.846250 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.847337 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.847758 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.850010 (+ 2252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:25.850250 (+ 240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:25.850254 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:25.850257 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:25.850280 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:25.852196 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:25.852735 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:25.852739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:25.852741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:25.852835 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:25.853160 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:25.853164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:25.853165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:25.853373 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:25.854288 (+ 915us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:25.854373 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 14:59:25.854499 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":138,"threads_started":1}
I20260927 14:59:25.856865 13762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74: Neither blocks nor log segments found. Creating new log.
Sep 27 14:59:25 dist-test-slave-pz0h krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:59:25.857873 13762 log.cc:824] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74: Log is configured to *not* fsync() on all Append() calls
I20260927 14:59:25.858093 13837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30f235a41d0d4533a052bfd6c144e453: Bootstrap starting.
Sep 27 14:59:25 dist-test-slave-pz0h krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 14:59:25.861291 13837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30f235a41d0d4533a052bfd6c144e453: Neither blocks nor log segments found. Creating new log.
I20260927 14:59:25.862275 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.851470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56730 (local address 127.0.98.62:42131)
0927 14:59:25.851590 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:25.851594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:25.851682 (+ 88us) server_negotiation.cc:407] Connection header received
0927 14:59:25.851859 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:25.851862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:25.851910 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:25.851962 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:25.852803 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.853508 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.854847 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.855148 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.855220 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:25.858105 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:25.858110 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:25.858113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:25.858149 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:25.860128 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:25.860693 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:25.860697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:25.860699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:25.860793 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:25.861115 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:25.861119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:25.861120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:25.861368 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:25.861570 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:25.861956 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 14:59:25.862110 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 14:59:25.862301 13837 log.cc:824] T 00000000000000000000000000000000 P 30f235a41d0d4533a052bfd6c144e453: Log is configured to *not* fsync() on all Append() calls
I20260927 14:59:25.865077 13837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30f235a41d0d4533a052bfd6c144e453: No bootstrap required, opened a new log
I20260927 14:59:25.866276 13762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74: No bootstrap required, opened a new log
I20260927 14:59:25.866969 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.850320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52448 (local address 127.0.98.61:41181)
0927 14:59:25.850441 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:25.850445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:25.850693 (+ 248us) server_negotiation.cc:407] Connection header received
0927 14:59:25.850904 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:25.850907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:25.850957 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:25.851026 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:25.851559 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.852279 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.853340 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.853764 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.853847 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:25.862395 (+ 8548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:25.862423 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:25.862427 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:25.862459 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:25.864853 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:25.865408 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:25.865412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:25.865414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:25.865512 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:25.865856 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:25.865860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:25.865861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:25.866146 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:25.866226 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:25.866610 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 14:59:25.866731 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 14:59:25.867419 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.850242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41181 (local address 127.0.0.1:52448)
0927 14:59:25.850624 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:25.850638 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:25.850761 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:25.851149 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:25.851152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:25.851312 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:59:25.851430 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:25.851438 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.852419 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.852422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:25.853209 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:25.853219 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.854006 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.854010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:25.854127 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:25.858479 (+ 4352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:59:25.858496 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:59:25.862155 (+ 3659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:59:25.865003 (+ 2848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:25.865007 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:25.865010 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:25.865297 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:25.865648 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:25.865651 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:25.865653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:25.865748 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:25.866306 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:59:25.866309 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:59:25.866425 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 14:59:25.867104 (+ 679us) client_negotiation.cc:240] Negotiation successful
0927 14:59:25.867263 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297}
I20260927 14:59:25.868347 13913 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34417 } has no permanent_uuid. Determining permanent_uuid...
I20260927 14:59:25.870455 13837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30f235a41d0d4533a052bfd6c144e453 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d415904c59af4d8eb27cb8d67307ef74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42131 } } peers { permanent_uuid: "30f235a41d0d4533a052bfd6c144e453" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "51780b49284348a7a7ccf9d6ae034c72" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34417 } }
Sep 27 14:59:25 dist-test-slave-pz0h krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 14:59:25.870924 13837 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30f235a41d0d4533a052bfd6c144e453 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:59:25.871037 13837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30f235a41d0d4533a052bfd6c144e453 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f235a41d0d4533a052bfd6c144e453, State: Initialized, Role: FOLLOWER
I20260927 14:59:25.871037 13762 raft_consensus.cc:348] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d415904c59af4d8eb27cb8d67307ef74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42131 } } peers { permanent_uuid: "30f235a41d0d4533a052bfd6c144e453" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "51780b49284348a7a7ccf9d6ae034c72" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34417 } }
I20260927 14:59:25.871331 13762 raft_consensus.cc:374] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:59:25.871419 13762 raft_consensus.cc:729] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d415904c59af4d8eb27cb8d67307ef74, State: Initialized, Role: FOLLOWER
I20260927 14:59:25.871703 13837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30f235a41d0d4533a052bfd6c144e453 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d415904c59af4d8eb27cb8d67307ef74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42131 } } peers { permanent_uuid: "30f235a41d0d4533a052bfd6c144e453" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "51780b49284348a7a7ccf9d6ae034c72" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34417 } }
I20260927 14:59:25.871892 13762 consensus_queue.cc:260] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d415904c59af4d8eb27cb8d67307ef74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42131 } } peers { permanent_uuid: "30f235a41d0d4533a052bfd6c144e453" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "51780b49284348a7a7ccf9d6ae034c72" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34417 } }
I20260927 14:59:25.873401 13837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30f235a41d0d4533a052bfd6c144e453 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:59:25.874009 13762 sys_catalog.cc:564] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74 [sys.catalog]: configured and running, proceeding with master startup.
Sep 27 14:59:25 dist-test-slave-pz0h krb5kdc[13681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 14:59:25.878604 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.868930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51628 (local address 127.0.98.60:34417)
0927 14:59:25.869041 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:25.869045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:25.869107 (+ 62us) server_negotiation.cc:407] Connection header received
0927 14:59:25.869218 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:25.869221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:25.869260 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:25.869321 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:25.870616 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.871388 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.872442 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.872793 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.872886 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:25.875299 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:25.875303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:25.875306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:25.875332 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:25.876927 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:25.877382 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:25.877385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:25.877387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:25.877467 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:25.877738 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:25.877740 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:25.877742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:25.877950 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:25.878013 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:25.878295 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 14:59:25.878437 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 14:59:25.879257 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.868869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34417 (local address 127.0.0.1:51628)
0927 14:59:25.869038 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:25.869051 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:25.869135 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:25.869331 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:25.869334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:25.870357 (+ 1023us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:59:25.870471 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:25.870479 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.871498 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.871501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:25.872301 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:25.872315 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.873045 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.873048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:25.873131 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:25.873562 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:59:25.873576 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:59:25.875149 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:59:25.877049 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:25.877051 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:25.877054 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:25.877290 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:25.877558 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:25.877560 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:25.877562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:25.877640 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:25.878066 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:59:25.878069 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:59:25.878140 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 14:59:25.878968 (+ 828us) client_negotiation.cc:240] Negotiation successful
0927 14:59:25.879113 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
W20260927 14:59:25.880443 13939 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 30f235a41d0d4533a052bfd6c144e453: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:59:25.880864 13939 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:59:25.880708 13922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30f235a41d0d4533a052bfd6c144e453 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d415904c59af4d8eb27cb8d67307ef74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42131 } } peers { permanent_uuid: "30f235a41d0d4533a052bfd6c144e453" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "51780b49284348a7a7ccf9d6ae034c72" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34417 } } }
I20260927 14:59:25.881165 13922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30f235a41d0d4533a052bfd6c144e453 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:59:25.877449 13768 master_runner.cc:186] Error getting master registration for 127.0.98.62:42131: OK, instance_id { permanent_uuid: "d415904c59af4d8eb27cb8d67307ef74" instance_seqno: 1790521165362223 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 14:59:25.881538 13768 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 14:59:25.875882 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.865321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52458 (local address 127.0.98.61:41181)
0927 14:59:25.865437 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:25.865446 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:25.865457 (+ 11us) server_negotiation.cc:407] Connection header received
0927 14:59:25.865488 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:25.865490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:25.865531 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:25.865601 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:25.866452 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.867266 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.868281 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.868573 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.868668 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:25.871529 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:25.871533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:25.871536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:25.871565 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:25.873381 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:25.873919 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:25.873923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:25.873925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:25.874017 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:25.874338 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:25.874342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:25.874344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:25.874589 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:25.875481 (+ 892us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:25.875567 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 14:59:25.875703 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:59:25.882611 13913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51780b49284348a7a7ccf9d6ae034c72: Bootstrap starting.
I20260927 14:59:25.885375 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:59:25.885584 13913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51780b49284348a7a7ccf9d6ae034c72: Neither blocks nor log segments found. Creating new log.
W20260927 14:59:25.886096 13945 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:59:25.886183 13945 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:59:25.886499 13913 log.cc:824] T 00000000000000000000000000000000 P 51780b49284348a7a7ccf9d6ae034c72: Log is configured to *not* fsync() on all Append() calls
I20260927 14:59:25.883638 13694 master_runner.cc:186] Error getting master registration for 127.0.98.60:34417: OK, instance_id { permanent_uuid: "51780b49284348a7a7ccf9d6ae034c72" instance_seqno: 1790521165821672 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 14:59:25.887609 13694 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 14:59:25.891065 13923 sys_catalog.cc:455] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d415904c59af4d8eb27cb8d67307ef74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42131 } } peers { permanent_uuid: "30f235a41d0d4533a052bfd6c144e453" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "51780b49284348a7a7ccf9d6ae034c72" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34417 } } }
I20260927 14:59:25.891251 13923 sys_catalog.cc:458] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:59:25.898753 13913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51780b49284348a7a7ccf9d6ae034c72: No bootstrap required, opened a new log
I20260927 14:59:25.904137 13913 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51780b49284348a7a7ccf9d6ae034c72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d415904c59af4d8eb27cb8d67307ef74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42131 } } peers { permanent_uuid: "30f235a41d0d4533a052bfd6c144e453" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "51780b49284348a7a7ccf9d6ae034c72" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34417 } }
I20260927 14:59:25.904599 13913 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51780b49284348a7a7ccf9d6ae034c72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:59:25.904724 13913 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51780b49284348a7a7ccf9d6ae034c72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51780b49284348a7a7ccf9d6ae034c72, State: Initialized, Role: FOLLOWER
I20260927 14:59:25.905380 13913 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51780b49284348a7a7ccf9d6ae034c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d415904c59af4d8eb27cb8d67307ef74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42131 } } peers { permanent_uuid: "30f235a41d0d4533a052bfd6c144e453" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "51780b49284348a7a7ccf9d6ae034c72" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34417 } }
I20260927 14:59:25.907131 13913 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51780b49284348a7a7ccf9d6ae034c72 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:59:25.907711 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.895719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52464 (local address 127.0.98.61:41181)
0927 14:59:25.895841 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:25.895845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:25.896070 (+ 225us) server_negotiation.cc:407] Connection header received
0927 14:59:25.896298 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:25.896302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:25.896359 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:25.896419 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:25.897414 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.898172 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.899307 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.899641 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.899720 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:25.901651 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:25.901655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:25.901658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:25.901690 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:25.903566 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:25.904833 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:25.904837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:25.904839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:25.904938 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:25.906273 (+ 1335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:25.906278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:25.906279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:25.906527 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:25.906650 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:25.907406 (+ 756us) server_negotiation.cc:299] Negotiation successful
0927 14:59:25.907552 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 14:59:25.918313 13844 master_runner.cc:263] No leader master found from master 51780b49284348a7a7ccf9d6ae034c72
I20260927 14:59:25.918393 13844 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 14:59:25.920352 13967 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 51780b49284348a7a7ccf9d6ae034c72: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 14:59:25.922271 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.895380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56734 (local address 127.0.98.62:42131)
0927 14:59:25.895512 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:25.895516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:25.900504 (+ 4988us) server_negotiation.cc:407] Connection header received
0927 14:59:25.902567 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:25.902572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:25.902645 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:25.902710 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:25.908312 (+ 5602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.909040 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.911973 (+ 2933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.912291 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.912369 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:25.916431 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:25.916435 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:25.916438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:25.916474 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:25.918726 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:25.919604 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:25.919608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:25.919609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:25.919705 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:25.920434 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:25.920438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:25.920440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:25.920707 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:25.920841 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:25.921935 (+ 1094us) server_negotiation.cc:299] Negotiation successful
0927 14:59:25.922087 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 14:59:25.925203 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:25.896120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51636 (local address 127.0.98.60:34417)
0927 14:59:25.896235 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:25.896239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:25.904751 (+ 8512us) server_negotiation.cc:407] Connection header received
0927 14:59:25.905370 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:25.905374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:25.905428 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:25.905476 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:25.910202 (+ 4726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.910939 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.912981 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:25.913319 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:25.913395 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:25.916005 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:25.916009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:25.916012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:25.916042 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:25.917917 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:25.920012 (+ 2095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:25.920017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:25.920019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:25.920120 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:25.921198 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:25.921203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:25.921204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:25.921430 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:25.921560 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:25.924852 (+ 3292us) server_negotiation.cc:299] Negotiation successful
0927 14:59:25.925020 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 14:59:25.926306 13956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51780b49284348a7a7ccf9d6ae034c72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d415904c59af4d8eb27cb8d67307ef74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42131 } } peers { permanent_uuid: "30f235a41d0d4533a052bfd6c144e453" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "51780b49284348a7a7ccf9d6ae034c72" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34417 } } }
I20260927 14:59:25.926551 13956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51780b49284348a7a7ccf9d6ae034c72 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 14:59:25.926477 13967 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:59:25.956883 13923 raft_consensus.cc:482] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:59:25.957100 13923 raft_consensus.cc:504] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d415904c59af4d8eb27cb8d67307ef74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42131 } } peers { permanent_uuid: "30f235a41d0d4533a052bfd6c144e453" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "51780b49284348a7a7ccf9d6ae034c72" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34417 } }
I20260927 14:59:25.958225 13923 leader_election.cc:288] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30f235a41d0d4533a052bfd6c144e453 (127.0.98.61:41181), 51780b49284348a7a7ccf9d6ae034c72 (127.0.98.60:34417)
I20260927 14:59:25.959208 13887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d415904c59af4d8eb27cb8d67307ef74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51780b49284348a7a7ccf9d6ae034c72" is_pre_election: true
I20260927 14:59:25.959606 13887 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 51780b49284348a7a7ccf9d6ae034c72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d415904c59af4d8eb27cb8d67307ef74 in term 0.
I20260927 14:59:25.960316 13713 leader_election.cc:302] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51780b49284348a7a7ccf9d6ae034c72, d415904c59af4d8eb27cb8d67307ef74; no voters:
I20260927 14:59:25.961297 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d415904c59af4d8eb27cb8d67307ef74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f235a41d0d4533a052bfd6c144e453" is_pre_election: true
I20260927 14:59:25.961625 13811 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 30f235a41d0d4533a052bfd6c144e453 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d415904c59af4d8eb27cb8d67307ef74 in term 0.
I20260927 14:59:25.963040 13923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:59:25.963147 13923 raft_consensus.cc:482] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:59:25.963188 13923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:59:25.964576 13923 raft_consensus.cc:504] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d415904c59af4d8eb27cb8d67307ef74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42131 } } peers { permanent_uuid: "30f235a41d0d4533a052bfd6c144e453" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "51780b49284348a7a7ccf9d6ae034c72" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34417 } }
I20260927 14:59:25.964964 13923 leader_election.cc:288] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74 [CANDIDATE]: Term 1 election: Requested vote from peers 30f235a41d0d4533a052bfd6c144e453 (127.0.98.61:41181), 51780b49284348a7a7ccf9d6ae034c72 (127.0.98.60:34417)
I20260927 14:59:25.965837 13811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d415904c59af4d8eb27cb8d67307ef74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f235a41d0d4533a052bfd6c144e453"
I20260927 14:59:25.965971 13811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30f235a41d0d4533a052bfd6c144e453 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:59:25.967335 13811 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 30f235a41d0d4533a052bfd6c144e453 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d415904c59af4d8eb27cb8d67307ef74 in term 1.
I20260927 14:59:25.967778 13712 leader_election.cc:302] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30f235a41d0d4533a052bfd6c144e453, d415904c59af4d8eb27cb8d67307ef74; no voters:
I20260927 14:59:25.968094 13887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d415904c59af4d8eb27cb8d67307ef74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51780b49284348a7a7ccf9d6ae034c72"
I20260927 14:59:25.968231 13887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51780b49284348a7a7ccf9d6ae034c72 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:59:25.969509 13887 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 51780b49284348a7a7ccf9d6ae034c72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d415904c59af4d8eb27cb8d67307ef74 in term 1.
I20260927 14:59:25.972640 13923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:59:25.972965 13923 raft_consensus.cc:686] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74 [term 1 LEADER]: Becoming Leader. State: Replica: d415904c59af4d8eb27cb8d67307ef74, State: Running, Role: LEADER
I20260927 14:59:25.973394 13923 consensus_queue.cc:237] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d415904c59af4d8eb27cb8d67307ef74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42131 } } peers { permanent_uuid: "30f235a41d0d4533a052bfd6c144e453" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "51780b49284348a7a7ccf9d6ae034c72" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34417 } }
I20260927 14:59:25.975418 13923 sys_catalog.cc:455] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d415904c59af4d8eb27cb8d67307ef74. Latest consensus state: current_term: 1 leader_uuid: "d415904c59af4d8eb27cb8d67307ef74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d415904c59af4d8eb27cb8d67307ef74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42131 } } peers { permanent_uuid: "30f235a41d0d4533a052bfd6c144e453" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "51780b49284348a7a7ccf9d6ae034c72" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34417 } } }
I20260927 14:59:25.975610 13923 sys_catalog.cc:458] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74 [sys.catalog]: This master's current role is: LEADER
I20260927 14:59:25.976050 13971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:59:25.977186 13971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:59:25.988713 13887 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 51780b49284348a7a7ccf9d6ae034c72 [term 1 FOLLOWER]: Refusing update from remote peer d415904c59af4d8eb27cb8d67307ef74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:59:25.989643 13923 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51780b49284348a7a7ccf9d6ae034c72" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:59:25.994724 13956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51780b49284348a7a7ccf9d6ae034c72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d415904c59af4d8eb27cb8d67307ef74. Latest consensus state: current_term: 1 leader_uuid: "d415904c59af4d8eb27cb8d67307ef74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d415904c59af4d8eb27cb8d67307ef74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42131 } } peers { permanent_uuid: "30f235a41d0d4533a052bfd6c144e453" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "51780b49284348a7a7ccf9d6ae034c72" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34417 } } }
I20260927 14:59:25.994922 13956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51780b49284348a7a7ccf9d6ae034c72 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:59:25.999279 13811 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 30f235a41d0d4533a052bfd6c144e453 [term 1 FOLLOWER]: Refusing update from remote peer d415904c59af4d8eb27cb8d67307ef74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:59:26.000072 13923 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30f235a41d0d4533a052bfd6c144e453" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:59:26.004837 13922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30f235a41d0d4533a052bfd6c144e453 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d415904c59af4d8eb27cb8d67307ef74. Latest consensus state: current_term: 1 leader_uuid: "d415904c59af4d8eb27cb8d67307ef74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d415904c59af4d8eb27cb8d67307ef74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42131 } } peers { permanent_uuid: "30f235a41d0d4533a052bfd6c144e453" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "51780b49284348a7a7ccf9d6ae034c72" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34417 } } }
I20260927 14:59:26.005000 13922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30f235a41d0d4533a052bfd6c144e453 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:59:26.020458 13969 sys_catalog.cc:455] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d415904c59af4d8eb27cb8d67307ef74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d415904c59af4d8eb27cb8d67307ef74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42131 } } peers { permanent_uuid: "30f235a41d0d4533a052bfd6c144e453" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "51780b49284348a7a7ccf9d6ae034c72" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34417 } } }
I20260927 14:59:26.020632 13969 sys_catalog.cc:458] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74 [sys.catalog]: This master's current role is: LEADER
I20260927 14:59:26.022047 13922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30f235a41d0d4533a052bfd6c144e453 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d415904c59af4d8eb27cb8d67307ef74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d415904c59af4d8eb27cb8d67307ef74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42131 } } peers { permanent_uuid: "30f235a41d0d4533a052bfd6c144e453" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "51780b49284348a7a7ccf9d6ae034c72" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34417 } } }
I20260927 14:59:26.022186 13922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30f235a41d0d4533a052bfd6c144e453 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:59:26.022696 13972 mvcc.cc:205] Tried to move back new op lower bound from 7333974695880314880 to 7333974695830925312. Current Snapshot: MvccSnapshot[applied={T|T < 7333974695880314880}]
I20260927 14:59:26.025278 13923 sys_catalog.cc:455] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d415904c59af4d8eb27cb8d67307ef74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d415904c59af4d8eb27cb8d67307ef74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42131 } } peers { permanent_uuid: "30f235a41d0d4533a052bfd6c144e453" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "51780b49284348a7a7ccf9d6ae034c72" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34417 } } }
I20260927 14:59:26.025429 13923 sys_catalog.cc:458] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74 [sys.catalog]: This master's current role is: LEADER
I20260927 14:59:26.027446 13971 catalog_manager.cc:1347] Generated new cluster ID: ed94910a00634475b7aa286fcb2a56c8
I20260927 14:59:26.027513 13971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:59:26.032917 13956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51780b49284348a7a7ccf9d6ae034c72 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d415904c59af4d8eb27cb8d67307ef74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d415904c59af4d8eb27cb8d67307ef74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42131 } } peers { permanent_uuid: "30f235a41d0d4533a052bfd6c144e453" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "51780b49284348a7a7ccf9d6ae034c72" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34417 } } }
I20260927 14:59:26.033102 13956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51780b49284348a7a7ccf9d6ae034c72 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 14:59:26.041409 13971 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:59:26.042814 13971 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:59:26.024219 13975 mvcc.cc:205] Tried to move back new op lower bound from 7333974695880314880 to 7333974695830925312. Current Snapshot: MvccSnapshot[applied={T|T < 7333974695880314880}]
I20260927 14:59:26.054909 13973 mvcc.cc:205] Tried to move back new op lower bound from 7333974695880314880 to 7333974695830925312. Current Snapshot: MvccSnapshot[applied={T|T < 7333974695880314880}]
I20260927 14:59:26.070363 13971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d415904c59af4d8eb27cb8d67307ef74: Generated new TSK 0
I20260927 14:59:26.071446 13971 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:59:26.882278 13939 catalog_manager.cc:1259] Loaded cluster ID: ed94910a00634475b7aa286fcb2a56c8
I20260927 14:59:26.882375 13939 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 30f235a41d0d4533a052bfd6c144e453: loading cluster ID for follower catalog manager: success
I20260927 14:59:26.884159 13939 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 30f235a41d0d4533a052bfd6c144e453: acquiring CA information for follower catalog manager: success
I20260927 14:59:26.885421 13939 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 30f235a41d0d4533a052bfd6c144e453: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 14:59:26.928117 13967 catalog_manager.cc:1259] Loaded cluster ID: ed94910a00634475b7aa286fcb2a56c8
I20260927 14:59:26.928226 13967 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 51780b49284348a7a7ccf9d6ae034c72: loading cluster ID for follower catalog manager: success
I20260927 14:59:26.929943 13967 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 51780b49284348a7a7ccf9d6ae034c72: acquiring CA information for follower catalog manager: success
I20260927 14:59:26.931227 13967 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 51780b49284348a7a7ccf9d6ae034c72: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 14:59:27.139130 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 13694
I20260927 14:59:27.145090 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 13768
I20260927 14:59:27.150511 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 13844
2026-09-27T14:59:27Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3050 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:59:27 dist-test-slave-pz0h krb5kdc[13988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:59:27 dist-test-slave-pz0h krb5kdc[13988](info): set up 2 sockets
Sep 27 14:59:27 dist-test-slave-pz0h krb5kdc[13988](info): commencing operation
krb5kdc: starting...
W20260927 14:59:28.110867 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 14:59:28 dist-test-slave-pz0h krb5kdc[13988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:59:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:59:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:59:28.229391 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38041
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38041
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:59:28.343932 14001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:59:28.344247 14001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:59:28.344300 14001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:59:28.348529 14001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:59:28.348619 14001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:59:28.348639 14001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:59:28.348658 14001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:59:28.348672 14001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:59:28.354543 14001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:59:28.355751 14001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:59:28.356774 14001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:59:28.365296 14006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:28.365296 14007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:28.365296 14009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:28.365386 14001 server_base.cc:1034] running on GCE node
I20260927 14:59:28.366218 14001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:59:28.367465 14001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:59:28.368643 14001 hybrid_clock.cc:648] HybridClock initialized: now 1790521168368616 us; error 40 us; skew 500 ppm
Sep 27 14:59:28 dist-test-slave-pz0h krb5kdc[13988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:59:28.371555 14001 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:59:28.372601 14001 webserver.cc:460] Webserver started at http://127.0.98.62:42847/ using document root <none> and password file <none>
I20260927 14:59:28.373186 14001 fs_manager.cc:359] Metadata directory not provided
I20260927 14:59:28.373260 14001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:59:28.373471 14001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:59:28.375712 14001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "c4ebf9bf90c14a33b7d943f98f499d6b"
format_stamp: "Formatted at 2026-09-27 14:59:28 on dist-test-slave-pz0h"
server_key: "150d2729db2e1459b075a4837120cf39"
server_key_iv: "5c0f48e4b3eabfe4cdab2934b4916877"
server_key_version: "encryptionkey@0"
I20260927 14:59:28.376341 14001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "c4ebf9bf90c14a33b7d943f98f499d6b"
format_stamp: "Formatted at 2026-09-27 14:59:28 on dist-test-slave-pz0h"
server_key: "150d2729db2e1459b075a4837120cf39"
server_key_iv: "5c0f48e4b3eabfe4cdab2934b4916877"
server_key_version: "encryptionkey@0"
I20260927 14:59:28.380627 14001 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 14:59:28.383466 14016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:28.385030 14001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:59:28.385210 14001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "c4ebf9bf90c14a33b7d943f98f499d6b"
format_stamp: "Formatted at 2026-09-27 14:59:28 on dist-test-slave-pz0h"
server_key: "150d2729db2e1459b075a4837120cf39"
server_key_iv: "5c0f48e4b3eabfe4cdab2934b4916877"
server_key_version: "encryptionkey@0"
I20260927 14:59:28.385346 14001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:59:28.404038 14001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:59:28.408550 14001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:59:28.408799 14001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:59:28.416357 14001 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38041
I20260927 14:59:28.416359 14068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38041 every 8 connection(s)
I20260927 14:59:28.417479 14001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:59:28.420692 14069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:59:28.426576 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 14001
I20260927 14:59:28.426729 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 14:59:28.427033 392 external_mini_cluster.cc:1428] Setting key 3f270d03f1043e739a5f8ea95b0ae513
I20260927 14:59:28.427356 14069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4ebf9bf90c14a33b7d943f98f499d6b: Bootstrap starting.
I20260927 14:59:28.430061 14069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4ebf9bf90c14a33b7d943f98f499d6b: Neither blocks nor log segments found. Creating new log.
I20260927 14:59:28.430876 14069 log.cc:824] T 00000000000000000000000000000000 P c4ebf9bf90c14a33b7d943f98f499d6b: Log is configured to *not* fsync() on all Append() calls
I20260927 14:59:28.433039 14069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4ebf9bf90c14a33b7d943f98f499d6b: No bootstrap required, opened a new log
Sep 27 14:59:28 dist-test-slave-pz0h krb5kdc[13988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:59:28.437774 14069 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4ebf9bf90c14a33b7d943f98f499d6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ebf9bf90c14a33b7d943f98f499d6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38041 } }
I20260927 14:59:28.438050 14069 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4ebf9bf90c14a33b7d943f98f499d6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:59:28.438092 14069 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4ebf9bf90c14a33b7d943f98f499d6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4ebf9bf90c14a33b7d943f98f499d6b, State: Initialized, Role: FOLLOWER
I20260927 14:59:28.438587 14069 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4ebf9bf90c14a33b7d943f98f499d6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ebf9bf90c14a33b7d943f98f499d6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38041 } }
I20260927 14:59:28.438740 14069 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4ebf9bf90c14a33b7d943f98f499d6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:59:28.438786 14069 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4ebf9bf90c14a33b7d943f98f499d6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:59:28.438916 14069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4ebf9bf90c14a33b7d943f98f499d6b [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:59:28.439888 14069 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4ebf9bf90c14a33b7d943f98f499d6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ebf9bf90c14a33b7d943f98f499d6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38041 } }
I20260927 14:59:28.440250 14069 leader_election.cc:302] T 00000000000000000000000000000000 P c4ebf9bf90c14a33b7d943f98f499d6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4ebf9bf90c14a33b7d943f98f499d6b; no voters:
I20260927 14:59:28.440587 14069 leader_election.cc:288] T 00000000000000000000000000000000 P c4ebf9bf90c14a33b7d943f98f499d6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:59:28.440682 14074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4ebf9bf90c14a33b7d943f98f499d6b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:59:28.440846 14074 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4ebf9bf90c14a33b7d943f98f499d6b [term 1 LEADER]: Becoming Leader. State: Replica: c4ebf9bf90c14a33b7d943f98f499d6b, State: Running, Role: LEADER
I20260927 14:59:28.441267 14074 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4ebf9bf90c14a33b7d943f98f499d6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ebf9bf90c14a33b7d943f98f499d6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38041 } }
I20260927 14:59:28.442004 14069 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4ebf9bf90c14a33b7d943f98f499d6b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:59:28.442994 14075 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4ebf9bf90c14a33b7d943f98f499d6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4ebf9bf90c14a33b7d943f98f499d6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ebf9bf90c14a33b7d943f98f499d6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38041 } } }
I20260927 14:59:28.443104 14072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:28.428491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46010 (local address 127.0.98.62:38041)
0927 14:59:28.429027 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:28.429043 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:28.429081 (+ 38us) server_negotiation.cc:407] Connection header received
0927 14:59:28.429877 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:28.429900 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:28.430315 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:28.430660 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:28.431540 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:28.432549 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:28.433475 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:28.433980 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:28.434145 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:28.437014 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:28.437020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:28.437038 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:28.437080 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:28.439359 (+ 2279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:28.439832 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:28.439838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:28.439843 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:28.439974 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:28.440293 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:28.440296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:28.440298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:28.440856 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:28.441679 (+ 823us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:28.441813 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 14:59:28.442071 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":183,"threads_started":1}
I20260927 14:59:28.443156 14075 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4ebf9bf90c14a33b7d943f98f499d6b [sys.catalog]: This master's current role is: LEADER
I20260927 14:59:28.443388 14075 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4ebf9bf90c14a33b7d943f98f499d6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4ebf9bf90c14a33b7d943f98f499d6b. Latest consensus state: current_term: 1 leader_uuid: "c4ebf9bf90c14a33b7d943f98f499d6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ebf9bf90c14a33b7d943f98f499d6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38041 } } }
I20260927 14:59:28.443468 14075 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4ebf9bf90c14a33b7d943f98f499d6b [sys.catalog]: This master's current role is: LEADER
I20260927 14:59:28.444675 14083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:59:28.447113 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 14:59:28.448256 14083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:59:28.455096 14083 catalog_manager.cc:1347] Generated new cluster ID: 5b40ed3be4314ad3a0a9df26e9bf9f52
I20260927 14:59:28.455205 14083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:59:28.474633 14083 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:59:28.475685 14083 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:59:28.480971 14083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4ebf9bf90c14a33b7d943f98f499d6b: Generated new TSK 0
I20260927 14:59:28.481707 14083 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 14:59:28.503677 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 14001
2026-09-27T14:59:28Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1352 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:59:28 dist-test-slave-pz0h krb5kdc[14096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:59:28 dist-test-slave-pz0h krb5kdc[14096](info): set up 2 sockets
Sep 27 14:59:28 dist-test-slave-pz0h krb5kdc[14096](info): commencing operation
krb5kdc: starting...
W20260927 14:59:29.458876 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 14:59:29 dist-test-slave-pz0h krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:59:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:59:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:59:29.579661 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36713
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36713
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:59:29.710109 14109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:59:29.710419 14109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:59:29.710461 14109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:59:29.714687 14109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:59:29.714778 14109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:59:29.714818 14109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:59:29.714835 14109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:59:29.714849 14109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:59:29.720570 14109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:59:29.721750 14109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:59:29.722854 14109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:59:29.731909 14117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:29.731909 14115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:29.731948 14114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:29.732348 14109 server_base.cc:1034] running on GCE node
I20260927 14:59:29.732995 14109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:59:29.734138 14109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:59:29.735325 14109 hybrid_clock.cc:648] HybridClock initialized: now 1790521169735300 us; error 41 us; skew 500 ppm
Sep 27 14:59:29 dist-test-slave-pz0h krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:59:29.738188 14109 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:59:29.739225 14109 webserver.cc:460] Webserver started at http://127.0.98.62:32911/ using document root <none> and password file <none>
I20260927 14:59:29.739794 14109 fs_manager.cc:359] Metadata directory not provided
I20260927 14:59:29.739866 14109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:59:29.740082 14109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:59:29.742219 14109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "f11b988f649d47b2a387ac1ed39541ac"
format_stamp: "Formatted at 2026-09-27 14:59:29 on dist-test-slave-pz0h"
server_key: "e0eb297193e3ec910d1abb0116efa112"
server_key_iv: "4548a50933ae9567a46f8608573dfb77"
server_key_version: "encryptionkey@0"
I20260927 14:59:29.742852 14109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "f11b988f649d47b2a387ac1ed39541ac"
format_stamp: "Formatted at 2026-09-27 14:59:29 on dist-test-slave-pz0h"
server_key: "e0eb297193e3ec910d1abb0116efa112"
server_key_iv: "4548a50933ae9567a46f8608573dfb77"
server_key_version: "encryptionkey@0"
I20260927 14:59:29.746774 14109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260927 14:59:29.749315 14124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:29.750569 14109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 14:59:29.750699 14109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "f11b988f649d47b2a387ac1ed39541ac"
format_stamp: "Formatted at 2026-09-27 14:59:29 on dist-test-slave-pz0h"
server_key: "e0eb297193e3ec910d1abb0116efa112"
server_key_iv: "4548a50933ae9567a46f8608573dfb77"
server_key_version: "encryptionkey@0"
I20260927 14:59:29.750876 14109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:59:29.764410 14109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:59:29.767510 14109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:59:29.767702 14109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:59:29.774660 14109 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36713
I20260927 14:59:29.774946 14176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36713 every 8 connection(s)
I20260927 14:59:29.775701 14109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:59:29.776470 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 14109
I20260927 14:59:29.776607 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 14:59:29.777025 392 external_mini_cluster.cc:1428] Setting key cac1035bb9c9c6bb2730912b3cc58b38
I20260927 14:59:29.779224 14177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:59:29.785808 14177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac: Bootstrap starting.
Sep 27 14:59:29 dist-test-slave-pz0h krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:59:29.788601 14177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac: Neither blocks nor log segments found. Creating new log.
I20260927 14:59:29.789520 14177 log.cc:824] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac: Log is configured to *not* fsync() on all Append() calls
I20260927 14:59:29.791786 14180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:29.778399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42724 (local address 127.0.98.62:36713)
0927 14:59:29.778833 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:29.778853 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:29.778890 (+ 37us) server_negotiation.cc:407] Connection header received
0927 14:59:29.779567 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:29.779585 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:29.779940 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:29.780230 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:29.781063 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:29.782070 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:29.783226 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:29.783742 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:29.783869 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:29.786540 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:29.786547 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:29.786564 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:29.786599 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:29.788557 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:29.789050 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:29.789062 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:29.789069 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:29.789223 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:29.789453 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:29.789456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:29.789458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:29.789988 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:29.790149 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:29.790486 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 14:59:29.790829 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":108,"threads_started":1}
I20260927 14:59:29.792021 14177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac: No bootstrap required, opened a new log
I20260927 14:59:29.795969 14177 raft_consensus.cc:348] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11b988f649d47b2a387ac1ed39541ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36713 } }
I20260927 14:59:29.796231 14177 raft_consensus.cc:374] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:59:29.796274 14177 raft_consensus.cc:729] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f11b988f649d47b2a387ac1ed39541ac, State: Initialized, Role: FOLLOWER
I20260927 14:59:29.796674 14177 consensus_queue.cc:260] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11b988f649d47b2a387ac1ed39541ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36713 } }
I20260927 14:59:29.796789 14177 raft_consensus.cc:388] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:59:29.796836 14177 raft_consensus.cc:482] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:59:29.796907 14177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:59:29.797853 14177 raft_consensus.cc:504] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11b988f649d47b2a387ac1ed39541ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36713 } }
I20260927 14:59:29.798175 14177 leader_election.cc:302] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f11b988f649d47b2a387ac1ed39541ac; no voters:
I20260927 14:59:29.798475 14177 leader_election.cc:288] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:59:29.798636 14182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:59:29.798887 14182 raft_consensus.cc:686] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [term 1 LEADER]: Becoming Leader. State: Replica: f11b988f649d47b2a387ac1ed39541ac, State: Running, Role: LEADER
I20260927 14:59:29.799201 14182 consensus_queue.cc:237] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11b988f649d47b2a387ac1ed39541ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36713 } }
I20260927 14:59:29.799794 14177 sys_catalog.cc:564] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:59:29.800786 14184 sys_catalog.cc:455] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader f11b988f649d47b2a387ac1ed39541ac. Latest consensus state: current_term: 1 leader_uuid: "f11b988f649d47b2a387ac1ed39541ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11b988f649d47b2a387ac1ed39541ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36713 } } }
I20260927 14:59:29.800920 14184 sys_catalog.cc:458] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [sys.catalog]: This master's current role is: LEADER
I20260927 14:59:29.801244 14183 sys_catalog.cc:455] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f11b988f649d47b2a387ac1ed39541ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11b988f649d47b2a387ac1ed39541ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36713 } } }
I20260927 14:59:29.801311 14191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:59:29.801391 14183 sys_catalog.cc:458] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [sys.catalog]: This master's current role is: LEADER
I20260927 14:59:29.804937 14191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:59:29.811455 14191 catalog_manager.cc:1347] Generated new cluster ID: 6697c3923601418185e6da925f1a9680
I20260927 14:59:29.811558 14191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:59:29.849802 14191 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:59:29.850775 14191 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:59:29.862560 14191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac: Generated new TSK 0
I20260927 14:59:29.863379 14191 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:59:29.897341 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36713
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:59:30.014358 14203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:59:30.014683 14203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:59:30.014746 14203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:59:30.019461 14203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:59:30.019564 14203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:59:30.019656 14203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:59:30.025507 14203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:59:30.026667 14203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:59:30.028056 14203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:59:30.039733 14211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:30.039738 14208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:30.039738 14209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:30.039809 14203 server_base.cc:1034] running on GCE node
I20260927 14:59:30.040488 14203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:59:30.041182 14203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:59:30.042385 14203 hybrid_clock.cc:648] HybridClock initialized: now 1790521170042366 us; error 36 us; skew 500 ppm
Sep 27 14:59:30 dist-test-slave-pz0h krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:59:30.045382 14203 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:59:30.046339 14203 webserver.cc:460] Webserver started at http://127.0.98.1:40875/ using document root <none> and password file <none>
I20260927 14:59:30.046938 14203 fs_manager.cc:359] Metadata directory not provided
I20260927 14:59:30.047008 14203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:59:30.047212 14203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:59:30.049350 14203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/data/instance:
uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7"
format_stamp: "Formatted at 2026-09-27 14:59:30 on dist-test-slave-pz0h"
server_key: "a134d70a8a0419bf63d9b7f70c41695b"
server_key_iv: "50e7e9921f49a9a6e86abb51d92b283a"
server_key_version: "encryptionkey@0"
I20260927 14:59:30.049940 14203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7"
format_stamp: "Formatted at 2026-09-27 14:59:30 on dist-test-slave-pz0h"
server_key: "a134d70a8a0419bf63d9b7f70c41695b"
server_key_iv: "50e7e9921f49a9a6e86abb51d92b283a"
server_key_version: "encryptionkey@0"
I20260927 14:59:30.054143 14203 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 14:59:30.057255 14218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:30.059028 14203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 14:59:30.059173 14203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7"
format_stamp: "Formatted at 2026-09-27 14:59:30 on dist-test-slave-pz0h"
server_key: "a134d70a8a0419bf63d9b7f70c41695b"
server_key_iv: "50e7e9921f49a9a6e86abb51d92b283a"
server_key_version: "encryptionkey@0"
I20260927 14:59:30.059303 14203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:59:30.078346 14203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:59:30.082527 14203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:59:30.082728 14203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:59:30.083792 14203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:59:30.084981 14203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:59:30.085057 14203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:30.085103 14203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:59:30.085119 14203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:30.094482 14203 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40213
I20260927 14:59:30.094519 14331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40213 every 8 connection(s)
I20260927 14:59:30.095486 14203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:59:30.104875 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 14203
I20260927 14:59:30.104987 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:59:30.105249 392 external_mini_cluster.cc:1428] Setting key 8b1efd20a02e339549f39ddd266b4371
Sep 27 14:59:30 dist-test-slave-pz0h krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:59:30.110586 14180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:30.097446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46439 (local address 127.0.98.62:36713)
0927 14:59:30.097585 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:30.097589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:30.098734 (+ 1145us) server_negotiation.cc:407] Connection header received
0927 14:59:30.099752 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:30.099756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:30.099809 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:30.099885 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:30.101401 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.101968 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.103260 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.103520 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.103626 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:30.106370 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:30.106374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:30.106377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:30.106403 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:30.108064 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:30.108799 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:30.108804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:30.108806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:30.108919 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:30.109271 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:30.109274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:30.109275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:30.109524 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:30.109616 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:30.110253 (+ 637us) server_negotiation.cc:299] Negotiation successful
0927 14:59:30.110409 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 14:59:30.111475 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:30.097827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.1:46439)
0927 14:59:30.098572 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:30.098617 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:30.099542 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:30.100193 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:30.100205 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:30.100687 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:59:30.101230 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:30.101246 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.102106 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.102109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:30.103111 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:30.103126 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.103710 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.103714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:30.103890 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:30.104524 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:59:30.104541 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:59:30.106182 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:59:30.108235 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:30.108243 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:30.108265 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:30.108680 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:30.109016 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:30.109018 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:30.109020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:30.109178 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:30.109664 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:59:30.109671 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:59:30.109959 (+ 288us) client_negotiation.cc:769] Sending connection context
0927 14:59:30.110163 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 14:59:30.110535 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":119,"threads_started":1}
I20260927 14:59:30.112941 14332 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36713
I20260927 14:59:30.113186 14332 heartbeater.cc:464] Registering TS with master...
I20260927 14:59:30.113735 14332 heartbeater.cc:511] Master 127.0.98.62:36713 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 14:59:30.115697 14141 ts_manager.cc:194] Registered new tserver with Master: 648a2752f3ad4e51a06f1ee2db9ff2d7 (127.0.98.1:40213)
I20260927 14:59:30.117283 14141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46439
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:59:30.133677 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36713
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:59:30.252184 14337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:59:30.252512 14337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:59:30.252578 14337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:59:30.257084 14337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:59:30.257176 14337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:59:30.257261 14337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:59:30.262969 14337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:59:30.264171 14337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:59:30.265215 14337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 14:59:30.275126 14337 server_base.cc:1034] running on GCE node
W20260927 14:59:30.275085 14343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:30.275086 14345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:30.275071 14342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:30.275810 14337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:59:30.276471 14337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:59:30.277630 14337 hybrid_clock.cc:648] HybridClock initialized: now 1790521170277596 us; error 52 us; skew 500 ppm
Sep 27 14:59:30 dist-test-slave-pz0h krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:59:30.280287 14337 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:59:30.281193 14337 webserver.cc:460] Webserver started at http://127.0.98.2:33465/ using document root <none> and password file <none>
I20260927 14:59:30.281750 14337 fs_manager.cc:359] Metadata directory not provided
I20260927 14:59:30.281819 14337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:59:30.282020 14337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:59:30.284226 14337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/data/instance:
uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e"
format_stamp: "Formatted at 2026-09-27 14:59:30 on dist-test-slave-pz0h"
server_key: "dbb0fb17cc1a2ae96125889354d0b321"
server_key_iv: "568e28d849702a4dbf32fe404fd013df"
server_key_version: "encryptionkey@0"
I20260927 14:59:30.284807 14337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e"
format_stamp: "Formatted at 2026-09-27 14:59:30 on dist-test-slave-pz0h"
server_key: "dbb0fb17cc1a2ae96125889354d0b321"
server_key_iv: "568e28d849702a4dbf32fe404fd013df"
server_key_version: "encryptionkey@0"
I20260927 14:59:30.288780 14337 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 14:59:30.291616 14352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:30.292980 14337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:59:30.293128 14337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e"
format_stamp: "Formatted at 2026-09-27 14:59:30 on dist-test-slave-pz0h"
server_key: "dbb0fb17cc1a2ae96125889354d0b321"
server_key_iv: "568e28d849702a4dbf32fe404fd013df"
server_key_version: "encryptionkey@0"
I20260927 14:59:30.293251 14337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:59:30.306525 14337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:59:30.310163 14337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:59:30.310395 14337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:59:30.311121 14337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:59:30.312157 14337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:59:30.312216 14337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:30.312275 14337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:59:30.312309 14337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:30.321756 14337 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37381
I20260927 14:59:30.321776 14465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37381 every 8 connection(s)
I20260927 14:59:30.322773 14337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:59:30.330770 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 14337
I20260927 14:59:30.330936 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:59:30.331229 392 external_mini_cluster.cc:1428] Setting key f19ad13de63000c34b0fa2b97efa990b
Sep 27 14:59:30 dist-test-slave-pz0h krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:59:30.338873 14180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:30.324708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54419 (local address 127.0.98.62:36713)
0927 14:59:30.324891 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:30.324895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:30.325759 (+ 864us) server_negotiation.cc:407] Connection header received
0927 14:59:30.327045 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:30.327048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:30.327103 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:30.327190 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:30.328828 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.329378 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.330660 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.331039 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.331179 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:30.334043 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:30.334046 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:30.334049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:30.334084 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:30.336378 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:30.337045 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:30.337049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:30.337051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:30.337152 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:30.337556 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:30.337559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:30.337561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:30.337802 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:30.337894 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:30.338495 (+ 601us) server_negotiation.cc:299] Negotiation successful
0927 14:59:30.338651 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 14:59:30.339735 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:30.325131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.2:54419)
0927 14:59:30.325600 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:30.325645 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:30.326829 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:30.327400 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:30.327408 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:30.327911 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:59:30.328602 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:30.328627 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.329527 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.329530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:30.330490 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:30.330502 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.331287 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.331292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:30.331497 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:30.332132 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:59:30.332156 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:59:30.333845 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:59:30.336569 (+ 2724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:30.336576 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:30.336594 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:30.336929 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:30.337281 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:30.337284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:30.337285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:30.337459 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:30.337932 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:59:30.337939 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:59:30.338237 (+ 298us) client_negotiation.cc:769] Sending connection context
0927 14:59:30.338418 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 14:59:30.338728 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":105,"threads_started":1}
I20260927 14:59:30.341286 14466 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36713
I20260927 14:59:30.341569 14466 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 14:59:30.342638 14466 heartbeater.cc:511] Master 127.0.98.62:36713 requested a full tablet report, sending...
I20260927 14:59:30.344026 14141 ts_manager.cc:194] Registered new tserver with Master: e0f9c4b41077445fa0dc8cc9587f6f0e (127.0.98.2:37381)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 14:59:30.344709 14141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54419
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:59:30.359148 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36713
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:59:30.473737 14471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:59:30.474013 14471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:59:30.474058 14471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:59:30.478219 14471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:59:30.478307 14471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:59:30.478370 14471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:59:30.483908 14471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:59:30.485080 14471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:59:30.486150 14471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:59:30.496299 14476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:30.496299 14477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:30.496330 14479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:30.496366 14471 server_base.cc:1034] running on GCE node
I20260927 14:59:30.496987 14471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:59:30.497673 14471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:59:30.498883 14471 hybrid_clock.cc:648] HybridClock initialized: now 1790521170498861 us; error 58 us; skew 500 ppm
Sep 27 14:59:30 dist-test-slave-pz0h krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:59:30.501860 14471 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:59:30.502895 14471 webserver.cc:460] Webserver started at http://127.0.98.3:43251/ using document root <none> and password file <none>
I20260927 14:59:30.503475 14471 fs_manager.cc:359] Metadata directory not provided
I20260927 14:59:30.503547 14471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:59:30.503783 14471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:59:30.505909 14471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/data/instance:
uuid: "230effef1c7b4942827301a20af55add"
format_stamp: "Formatted at 2026-09-27 14:59:30 on dist-test-slave-pz0h"
server_key: "e6d493fc9f4e8cfd4c476291b0a4b9a9"
server_key_iv: "183b667df6b48df9119f533f5fedcd77"
server_key_version: "encryptionkey@0"
I20260927 14:59:30.506495 14471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "230effef1c7b4942827301a20af55add"
format_stamp: "Formatted at 2026-09-27 14:59:30 on dist-test-slave-pz0h"
server_key: "e6d493fc9f4e8cfd4c476291b0a4b9a9"
server_key_iv: "183b667df6b48df9119f533f5fedcd77"
server_key_version: "encryptionkey@0"
I20260927 14:59:30.510610 14471 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260927 14:59:30.513234 14486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:30.514534 14471 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 14:59:30.514657 14471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "230effef1c7b4942827301a20af55add"
format_stamp: "Formatted at 2026-09-27 14:59:30 on dist-test-slave-pz0h"
server_key: "e6d493fc9f4e8cfd4c476291b0a4b9a9"
server_key_iv: "183b667df6b48df9119f533f5fedcd77"
server_key_version: "encryptionkey@0"
I20260927 14:59:30.514773 14471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:59:30.547528 14471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:59:30.550755 14471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:59:30.550982 14471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:59:30.551594 14471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:59:30.552656 14471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:59:30.552712 14471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:30.552779 14471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:59:30.552805 14471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:30.561529 14471 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45817
I20260927 14:59:30.561549 14599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45817 every 8 connection(s)
I20260927 14:59:30.562487 14471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:59:30.566473 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 14471
I20260927 14:59:30.566586 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:59:30.566867 392 external_mini_cluster.cc:1428] Setting key ccfeb9d6b564a6d7666d48bb9a8e9383
Sep 27 14:59:30 dist-test-slave-pz0h krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:59:30.576458 14180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:30.564361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33301 (local address 127.0.98.62:36713)
0927 14:59:30.564505 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:30.564509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:30.565400 (+ 891us) server_negotiation.cc:407] Connection header received
0927 14:59:30.566724 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:30.566728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:30.566813 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:30.566893 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:30.568270 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.568800 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.569995 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.570207 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.570288 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:30.572927 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:30.572930 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:30.572933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:30.572961 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:30.574394 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:30.574995 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:30.574998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:30.574999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:30.575082 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:30.575445 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:30.575446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:30.575448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:30.575631 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:30.575703 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:30.576213 (+ 510us) server_negotiation.cc:299] Negotiation successful
0927 14:59:30.576330 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 14:59:30.577390 14602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:30.564757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.3:33301)
0927 14:59:30.565232 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:30.565278 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:30.566445 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:30.567065 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:30.567073 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:30.567535 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:59:30.568128 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:30.568149 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.568929 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.568931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:30.569884 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:30.569896 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.570337 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.570340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:30.570485 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:30.571089 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:59:30.571106 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:59:30.572682 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:59:30.574528 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:30.574534 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:30.574549 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:30.574900 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:30.575191 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:30.575194 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:30.575196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:30.575359 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:30.575721 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:59:30.575727 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:59:30.575996 (+ 269us) client_negotiation.cc:769] Sending connection context
0927 14:59:30.576183 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 14:59:30.576470 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":112,"threads_started":1}
I20260927 14:59:30.578692 14600 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36713
I20260927 14:59:30.578979 14600 heartbeater.cc:464] Registering TS with master...
I20260927 14:59:30.579615 14600 heartbeater.cc:511] Master 127.0.98.62:36713 requested a full tablet report, sending...
I20260927 14:59:30.580987 14140 ts_manager.cc:194] Registered new tserver with Master: 230effef1c7b4942827301a20af55add (127.0.98.3:45817)
I20260927 14:59:30.581612 14140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33301
I20260927 14:59:30.593555 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:59:30.605638 14180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:30.595754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42730 (local address 127.0.98.62:36713)
0927 14:59:30.595901 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:30.595904 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:30.595998 (+ 94us) server_negotiation.cc:407] Connection header received
0927 14:59:30.596125 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:30.596129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:30.596175 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:30.596245 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:30.597026 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.597676 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.598769 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.599045 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.599132 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:30.601440 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:30.601445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:30.601448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:30.601511 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:30.603632 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:30.604206 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:30.604209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:30.604211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:30.604311 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:30.604592 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:30.604595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:30.604597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:30.604942 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:30.605056 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:30.605386 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 14:59:30.605500 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 14:59:30.608737 14140 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:59:30.611994 14140 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:59:30.627653 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:30.621863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36604 (local address 127.0.98.3:45817)
0927 14:59:30.622239 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:30.622266 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:30.622628 (+ 362us) server_negotiation.cc:407] Connection header received
0927 14:59:30.622730 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:30.622734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:30.622931 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:30.623073 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:59:30.623277 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.624144 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.626138 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.626925 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.627029 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:30.627217 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:30.627376 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 14:59:30.627484 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":90,"threads_started":1}
I20260927 14:59:30.627653 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:30.621457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45817 (local address 127.0.0.1:36604)
0927 14:59:30.622529 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:30.622542 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:30.622634 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:30.623056 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:30.623059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:30.623078 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:59:30.623183 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:30.623190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.624301 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.624304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:30.625970 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:30.625982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.627048 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.627051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:30.627178 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:30.627219 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 14:59:30.627335 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 14:59:30.627478 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1011,"mutex_wait_us":131,"thread_start_us":105,"threads_started":1}
I20260927 14:59:30.628806 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:30.620761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40213 (local address 127.0.0.1:49718)
0927 14:59:30.621689 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:30.621743 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:30.621901 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:30.622671 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:30.622677 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:30.622701 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:59:30.622871 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:30.622878 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.624254 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.624261 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:30.626121 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:30.626138 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.627457 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.627460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:30.627552 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:30.627576 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 14:59:30.627643 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 14:59:30.627719 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"mutex_wait_us":60,"thread_start_us":100,"threads_started":1}
I20260927 14:59:30.629084 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:30.621124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49718 (local address 127.0.98.1:40213)
0927 14:59:30.621477 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:30.621483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:30.622043 (+ 560us) server_negotiation.cc:407] Connection header received
0927 14:59:30.622133 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:30.622138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:30.622366 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:30.622504 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:59:30.622995 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.624120 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.626296 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.627141 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.627251 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:30.627391 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:30.628098 (+ 707us) server_negotiation.cc:299] Negotiation successful
0927 14:59:30.628928 (+ 830us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":88,"threads_started":1}
I20260927 14:59:30.629655 14613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:30.620862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41360 (local address 127.0.98.2:37381)
0927 14:59:30.621380 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:30.621392 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:30.622159 (+ 767us) server_negotiation.cc:407] Connection header received
0927 14:59:30.622325 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:30.622334 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:30.622761 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:30.623124 (+ 363us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:59:30.623930 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.625018 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.626568 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.628315 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.628526 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:30.628818 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:30.629394 (+ 576us) server_negotiation.cc:299] Negotiation successful
0927 14:59:30.629497 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":108,"threads_started":1}
I20260927 14:59:30.629760 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:30.620696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37381 (local address 127.0.0.1:41360)
0927 14:59:30.621694 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:30.621758 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:30.621954 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:30.623491 (+ 1537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:30.623495 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:30.623535 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:59:30.623763 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:30.623778 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.625169 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.625173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:30.626379 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:30.626389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.628947 (+ 2558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.628951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:30.629176 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:30.629226 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 14:59:30.629354 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 14:59:30.629515 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":30,"thread_start_us":106,"threads_started":1}
I20260927 14:59:30.631210 14534 tablet_service.cc:1467] Processing CreateTablet for tablet f3bdc605de0146ae96c0365358fc7837 (DEFAULT_TABLE table=test-table [id=c40e0db6645547dcb3bbefb57e45c695]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:59:30.631778 14266 tablet_service.cc:1467] Processing CreateTablet for tablet f3bdc605de0146ae96c0365358fc7837 (DEFAULT_TABLE table=test-table [id=c40e0db6645547dcb3bbefb57e45c695]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:59:30.632416 14534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3bdc605de0146ae96c0365358fc7837. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:59:30.632560 14266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3bdc605de0146ae96c0365358fc7837. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:59:30.632650 14400 tablet_service.cc:1467] Processing CreateTablet for tablet f3bdc605de0146ae96c0365358fc7837 (DEFAULT_TABLE table=test-table [id=c40e0db6645547dcb3bbefb57e45c695]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:59:30.633634 14400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3bdc605de0146ae96c0365358fc7837. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:59:30.638754 14617 tablet_bootstrap.cc:492] T f3bdc605de0146ae96c0365358fc7837 P 648a2752f3ad4e51a06f1ee2db9ff2d7: Bootstrap starting.
I20260927 14:59:30.641114 14619 tablet_bootstrap.cc:492] T f3bdc605de0146ae96c0365358fc7837 P 230effef1c7b4942827301a20af55add: Bootstrap starting.
I20260927 14:59:30.641191 14617 tablet_bootstrap.cc:654] T f3bdc605de0146ae96c0365358fc7837 P 648a2752f3ad4e51a06f1ee2db9ff2d7: Neither blocks nor log segments found. Creating new log.
I20260927 14:59:30.642015 14617 log.cc:824] T f3bdc605de0146ae96c0365358fc7837 P 648a2752f3ad4e51a06f1ee2db9ff2d7: Log is configured to *not* fsync() on all Append() calls
I20260927 14:59:30.643033 14618 tablet_bootstrap.cc:492] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e: Bootstrap starting.
I20260927 14:59:30.643573 14619 tablet_bootstrap.cc:654] T f3bdc605de0146ae96c0365358fc7837 P 230effef1c7b4942827301a20af55add: Neither blocks nor log segments found. Creating new log.
I20260927 14:59:30.644582 14617 tablet_bootstrap.cc:492] T f3bdc605de0146ae96c0365358fc7837 P 648a2752f3ad4e51a06f1ee2db9ff2d7: No bootstrap required, opened a new log
I20260927 14:59:30.644603 14619 log.cc:824] T f3bdc605de0146ae96c0365358fc7837 P 230effef1c7b4942827301a20af55add: Log is configured to *not* fsync() on all Append() calls
I20260927 14:59:30.644805 14617 ts_tablet_manager.cc:1397] T f3bdc605de0146ae96c0365358fc7837 P 648a2752f3ad4e51a06f1ee2db9ff2d7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 14:59:30.645771 14618 tablet_bootstrap.cc:654] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e: Neither blocks nor log segments found. Creating new log.
I20260927 14:59:30.646495 14619 tablet_bootstrap.cc:492] T f3bdc605de0146ae96c0365358fc7837 P 230effef1c7b4942827301a20af55add: No bootstrap required, opened a new log
I20260927 14:59:30.646708 14619 ts_tablet_manager.cc:1397] T f3bdc605de0146ae96c0365358fc7837 P 230effef1c7b4942827301a20af55add: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 14:59:30.646697 14618 log.cc:824] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e: Log is configured to *not* fsync() on all Append() calls
I20260927 14:59:30.649326 14617 raft_consensus.cc:348] T f3bdc605de0146ae96c0365358fc7837 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 } } peers { permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 } } peers { permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } }
I20260927 14:59:30.649677 14617 raft_consensus.cc:374] T f3bdc605de0146ae96c0365358fc7837 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:59:30.649751 14617 raft_consensus.cc:729] T f3bdc605de0146ae96c0365358fc7837 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 648a2752f3ad4e51a06f1ee2db9ff2d7, State: Initialized, Role: FOLLOWER
I20260927 14:59:30.649807 14618 tablet_bootstrap.cc:492] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e: No bootstrap required, opened a new log
I20260927 14:59:30.650041 14618 ts_tablet_manager.cc:1397] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260927 14:59:30.650038 14619 raft_consensus.cc:348] T f3bdc605de0146ae96c0365358fc7837 P 230effef1c7b4942827301a20af55add [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 } } peers { permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 } } peers { permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } }
I20260927 14:59:30.650285 14619 raft_consensus.cc:374] T f3bdc605de0146ae96c0365358fc7837 P 230effef1c7b4942827301a20af55add [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:59:30.650342 14619 raft_consensus.cc:729] T f3bdc605de0146ae96c0365358fc7837 P 230effef1c7b4942827301a20af55add [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 230effef1c7b4942827301a20af55add, State: Initialized, Role: FOLLOWER
I20260927 14:59:30.650313 14617 consensus_queue.cc:260] T f3bdc605de0146ae96c0365358fc7837 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 } } peers { permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 } } peers { permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } }
I20260927 14:59:30.650771 14619 consensus_queue.cc:260] T f3bdc605de0146ae96c0365358fc7837 P 230effef1c7b4942827301a20af55add [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 } } peers { permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 } } peers { permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } }
I20260927 14:59:30.651059 14332 heartbeater.cc:503] Master 127.0.98.62:36713 was elected leader, sending a full tablet report...
I20260927 14:59:30.651707 14619 ts_tablet_manager.cc:1428] T f3bdc605de0146ae96c0365358fc7837 P 230effef1c7b4942827301a20af55add: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260927 14:59:30.652047 14600 heartbeater.cc:503] Master 127.0.98.62:36713 was elected leader, sending a full tablet report...
I20260927 14:59:30.652536 14617 ts_tablet_manager.cc:1428] T f3bdc605de0146ae96c0365358fc7837 P 648a2752f3ad4e51a06f1ee2db9ff2d7: Time spent starting tablet: real 0.008s user 0.000s sys 0.008s
I20260927 14:59:30.653913 14618 raft_consensus.cc:348] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 } } peers { permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 } } peers { permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } }
I20260927 14:59:30.654213 14618 raft_consensus.cc:374] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:59:30.654279 14618 raft_consensus.cc:729] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0f9c4b41077445fa0dc8cc9587f6f0e, State: Initialized, Role: FOLLOWER
I20260927 14:59:30.654826 14618 consensus_queue.cc:260] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 } } peers { permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 } } peers { permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } }
I20260927 14:59:30.656064 14618 ts_tablet_manager.cc:1428] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 14:59:30.656406 14466 heartbeater.cc:503] Master 127.0.98.62:36713 was elected leader, sending a full tablet report...
I20260927 14:59:30.780203 14625 raft_consensus.cc:482] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:59:30.780422 14625 raft_consensus.cc:504] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 } } peers { permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 } } peers { permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } }
I20260927 14:59:30.781620 14625 leader_election.cc:288] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 648a2752f3ad4e51a06f1ee2db9ff2d7 (127.0.98.1:40213), 230effef1c7b4942827301a20af55add (127.0.98.3:45817)
I20260927 14:59:30.785485 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:30.781861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40213 (local address 127.0.98.2:60327)
0927 14:59:30.781985 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:30.782001 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:30.782093 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:30.782485 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:30.782488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:30.782508 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:59:30.782633 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:30.782640 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.783543 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.783546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:30.784386 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:30.784395 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.785058 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.785062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:30.785165 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:30.785184 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 14:59:30.785249 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 14:59:30.785337 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":30}
I20260927 14:59:30.785485 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:30.782003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60327 (local address 127.0.98.1:40213)
0927 14:59:30.782259 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:30.782264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:30.782279 (+ 15us) server_negotiation.cc:407] Connection header received
0927 14:59:30.782335 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:30.782338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:30.782385 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:30.782465 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:59:30.782748 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.783437 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.784509 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.784936 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.785033 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:30.785102 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:30.785283 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 14:59:30.785365 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260927 14:59:30.786345 14286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3bdc605de0146ae96c0365358fc7837" candidate_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" is_pre_election: true
I20260927 14:59:30.786712 14286 raft_consensus.cc:2454] T f3bdc605de0146ae96c0365358fc7837 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0f9c4b41077445fa0dc8cc9587f6f0e in term 0.
I20260927 14:59:30.786849 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:30.782253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57321 (local address 127.0.98.3:45817)
0927 14:59:30.782396 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:30.782400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:30.782614 (+ 214us) server_negotiation.cc:407] Connection header received
0927 14:59:30.782712 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:30.782715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:30.782761 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:30.782852 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:59:30.783212 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.783953 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.785540 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.786196 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.786296 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:30.786359 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:30.786589 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 14:59:30.786681 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 14:59:30.787232 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:30.782124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45817 (local address 127.0.98.2:57321)
0927 14:59:30.782520 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:30.782532 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:30.782619 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:30.782974 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:30.782977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:30.782995 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:59:30.783103 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:30.783109 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.784090 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.784094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:30.785392 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:30.785403 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.786332 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.786336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:30.786436 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:30.786460 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 14:59:30.787000 (+ 540us) client_negotiation.cc:240] Negotiation successful
0927 14:59:30.787093 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":26,"thread_start_us":147,"threads_started":1}
I20260927 14:59:30.787298 14354 leader_election.cc:302] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 648a2752f3ad4e51a06f1ee2db9ff2d7, e0f9c4b41077445fa0dc8cc9587f6f0e; no voters:
I20260927 14:59:30.787621 14625 raft_consensus.cc:2781] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:59:30.787707 14625 raft_consensus.cc:482] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:59:30.787783 14625 raft_consensus.cc:3037] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:59:30.788203 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3bdc605de0146ae96c0365358fc7837" candidate_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "230effef1c7b4942827301a20af55add" is_pre_election: true
I20260927 14:59:30.788548 14554 raft_consensus.cc:2454] T f3bdc605de0146ae96c0365358fc7837 P 230effef1c7b4942827301a20af55add [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0f9c4b41077445fa0dc8cc9587f6f0e in term 0.
I20260927 14:59:30.789137 14625 raft_consensus.cc:504] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 } } peers { permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 } } peers { permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } }
I20260927 14:59:30.789466 14625 leader_election.cc:288] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e [CANDIDATE]: Term 1 election: Requested vote from peers 648a2752f3ad4e51a06f1ee2db9ff2d7 (127.0.98.1:40213), 230effef1c7b4942827301a20af55add (127.0.98.3:45817)
I20260927 14:59:30.789855 14286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3bdc605de0146ae96c0365358fc7837" candidate_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7"
I20260927 14:59:30.789880 14554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3bdc605de0146ae96c0365358fc7837" candidate_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "230effef1c7b4942827301a20af55add"
I20260927 14:59:30.789994 14554 raft_consensus.cc:3037] T f3bdc605de0146ae96c0365358fc7837 P 230effef1c7b4942827301a20af55add [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:59:30.789994 14286 raft_consensus.cc:3037] T f3bdc605de0146ae96c0365358fc7837 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:59:30.791349 14286 raft_consensus.cc:2454] T f3bdc605de0146ae96c0365358fc7837 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0f9c4b41077445fa0dc8cc9587f6f0e in term 1.
I20260927 14:59:30.791349 14554 raft_consensus.cc:2454] T f3bdc605de0146ae96c0365358fc7837 P 230effef1c7b4942827301a20af55add [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0f9c4b41077445fa0dc8cc9587f6f0e in term 1.
I20260927 14:59:30.791669 14354 leader_election.cc:302] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 648a2752f3ad4e51a06f1ee2db9ff2d7, e0f9c4b41077445fa0dc8cc9587f6f0e; no voters:
I20260927 14:59:30.791873 14625 raft_consensus.cc:2781] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:59:30.792068 14625 raft_consensus.cc:686] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e [term 1 LEADER]: Becoming Leader. State: Replica: e0f9c4b41077445fa0dc8cc9587f6f0e, State: Running, Role: LEADER
I20260927 14:59:30.792405 14625 consensus_queue.cc:237] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 } } peers { permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 } } peers { permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } }
I20260927 14:59:30.795586 14140 catalog_manager.cc:5495] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e reported cstate change: term changed from 0 to 1, leader changed from <none> to e0f9c4b41077445fa0dc8cc9587f6f0e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } health_report { overall_health: UNKNOWN } } }
W20260927 14:59:30.813508 14601 tablet.cc:2369] T f3bdc605de0146ae96c0365358fc7837 P 230effef1c7b4942827301a20af55add: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:59:30.822785 14613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:30.819593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41364 (local address 127.0.98.2:37381)
0927 14:59:30.819707 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:30.819710 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:30.819724 (+ 14us) server_negotiation.cc:407] Connection header received
0927 14:59:30.819801 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:30.819803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:30.819856 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:30.819951 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:59:30.820135 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.820625 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.821408 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:30.821641 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:30.821724 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:30.822003 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:59:30.822453 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:59:30.822556 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:30.822605 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 14:59:30.822662 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 14:59:30.823971 14467 tablet.cc:2369] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:59:30.828761 14554 raft_consensus.cc:1261] T f3bdc605de0146ae96c0365358fc7837 P 230effef1c7b4942827301a20af55add [term 1 FOLLOWER]: Refusing update from remote peer e0f9c4b41077445fa0dc8cc9587f6f0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:59:30.828761 14286 raft_consensus.cc:1261] T f3bdc605de0146ae96c0365358fc7837 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 1 FOLLOWER]: Refusing update from remote peer e0f9c4b41077445fa0dc8cc9587f6f0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:59:30.829479 14625 consensus_queue.cc:1035] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:59:30.829703 14627 consensus_queue.cc:1035] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:59:30.839768 14631 mvcc.cc:205] Tried to move back new op lower bound from 7333974715707084800 to 7333974715568369664. Current Snapshot: MvccSnapshot[applied={T|T < 7333974715707084800}]
I20260927 14:59:30.847046 14141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:59:30.847265 14141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:59:30.850973 14141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac: Sending DeleteTablet for 3 replicas of tablet f3bdc605de0146ae96c0365358fc7837
I20260927 14:59:30.851953 14400 tablet_service.cc:1514] Processing DeleteTablet for tablet f3bdc605de0146ae96c0365358fc7837 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:59:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41360
I20260927 14:59:30.852188 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 14203
I20260927 14:59:30.852041 14266 tablet_service.cc:1514] Processing DeleteTablet for tablet f3bdc605de0146ae96c0365358fc7837 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:59:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49718
I20260927 14:59:30.852053 14534 tablet_service.cc:1514] Processing DeleteTablet for tablet f3bdc605de0146ae96c0365358fc7837 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:59:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36604
I20260927 14:59:30.853930 14641 tablet_replica.cc:330] T f3bdc605de0146ae96c0365358fc7837 P 230effef1c7b4942827301a20af55add: stopping tablet replica
I20260927 14:59:30.854425 14640 tablet_replica.cc:330] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e: stopping tablet replica
I20260927 14:59:30.854738 14641 raft_consensus.cc:2229] T f3bdc605de0146ae96c0365358fc7837 P 230effef1c7b4942827301a20af55add [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:59:30.855185 14641 raft_consensus.cc:2258] T f3bdc605de0146ae96c0365358fc7837 P 230effef1c7b4942827301a20af55add [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:59:30.855746 14640 raft_consensus.cc:2229] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:59:30.856210 14640 raft_consensus.cc:2258] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:59:30.857064 14641 ts_tablet_manager.cc:1905] T f3bdc605de0146ae96c0365358fc7837 P 230effef1c7b4942827301a20af55add: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:59:30.857678 14640 ts_tablet_manager.cc:1905] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:59:30.858928 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 14337
W20260927 14:59:30.858971 14126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40213 (error 108)
I20260927 14:59:30.861361 14641 ts_tablet_manager.cc:1918] T f3bdc605de0146ae96c0365358fc7837 P 230effef1c7b4942827301a20af55add: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:59:30.861496 14641 log.cc:1196] T f3bdc605de0146ae96c0365358fc7837 P 230effef1c7b4942827301a20af55add: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/wal/wals/f3bdc605de0146ae96c0365358fc7837
I20260927 14:59:30.861756 14641 ts_tablet_manager.cc:1939] T f3bdc605de0146ae96c0365358fc7837 P 230effef1c7b4942827301a20af55add: Deleting consensus metadata
I20260927 14:59:30.863180 14126 catalog_manager.cc:4841] TS 230effef1c7b4942827301a20af55add (127.0.98.3:45817): tablet f3bdc605de0146ae96c0365358fc7837 (table test-table [id=c40e0db6645547dcb3bbefb57e45c695]) successfully deleted
I20260927 14:59:30.863610 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:30.863001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40213 (local address 127.0.0.1:49726)
0927 14:59:30.863445 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:30.863513 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360}
W20260927 14:59:30.863783 14126 catalog_manager.cc:4584] TS 648a2752f3ad4e51a06f1ee2db9ff2d7 (127.0.98.1:40213): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f3bdc605de0146ae96c0365358fc7837: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40213: connect: Connection refused (error 111)
I20260927 14:59:30.867079 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 14471
I20260927 14:59:30.867288 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:30.867013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37381 (local address 127.0.0.1:41376)
0927 14:59:30.867142 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:30.867198 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 14:59:30.872493 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 14109
I20260927 14:59:30.878666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36713
--webserver_interface=127.0.98.62
--webserver_port=32911
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:59:30.991672 14643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:59:30.991966 14643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:59:30.992007 14643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:59:30.996201 14643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:59:30.996277 14643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:59:30.996296 14643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:59:30.996312 14643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:59:30.996335 14643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:59:31.001828 14643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=32911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:59:31.002938 14643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:59:31.003973 14643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:59:31.018317 14649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:31.018317 14652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:31.018369 14643 server_base.cc:1034] running on GCE node
W20260927 14:59:31.018317 14650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:31.019379 14643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:59:31.020606 14643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:59:31.021827 14643 hybrid_clock.cc:648] HybridClock initialized: now 1790521171021795 us; error 35 us; skew 500 ppm
Sep 27 14:59:31 dist-test-slave-pz0h krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:59:31.024881 14643 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:59:31.025938 14643 webserver.cc:460] Webserver started at http://127.0.98.62:32911/ using document root <none> and password file <none>
I20260927 14:59:31.026522 14643 fs_manager.cc:359] Metadata directory not provided
I20260927 14:59:31.026609 14643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:59:31.031433 14643 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:59:31.034278 14659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:31.035687 14643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:59:31.035828 14643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "f11b988f649d47b2a387ac1ed39541ac"
format_stamp: "Formatted at 2026-09-27 14:59:29 on dist-test-slave-pz0h"
server_key: "e0eb297193e3ec910d1abb0116efa112"
server_key_iv: "4548a50933ae9567a46f8608573dfb77"
server_key_version: "encryptionkey@0"
I20260927 14:59:31.036473 14643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:59:31.047349 14643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:59:31.050606 14643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:59:31.050834 14643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:59:31.058072 14643 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36713
I20260927 14:59:31.058326 14711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36713 every 8 connection(s)
I20260927 14:59:31.059267 14643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:59:31.063832 14712 sys_catalog.cc:263] Verifying existing consensus state
I20260927 14:59:31.065531 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 14643
I20260927 14:59:31.065550 14712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac: Bootstrap starting.
I20260927 14:59:31.065951 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40213
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36713
--webserver_port=40875
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:59:31.083009 14712 log.cc:824] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac: Log is configured to *not* fsync() on all Append() calls
I20260927 14:59:31.092006 14712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 14:59:31.092414 14712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac: Bootstrap complete.
I20260927 14:59:31.096534 14712 raft_consensus.cc:348] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11b988f649d47b2a387ac1ed39541ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36713 } }
I20260927 14:59:31.096966 14712 raft_consensus.cc:729] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f11b988f649d47b2a387ac1ed39541ac, State: Initialized, Role: FOLLOWER
I20260927 14:59:31.097494 14712 consensus_queue.cc:260] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11b988f649d47b2a387ac1ed39541ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36713 } }
I20260927 14:59:31.097620 14712 raft_consensus.cc:388] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:59:31.097690 14712 raft_consensus.cc:482] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:59:31.097781 14712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [term 1 FOLLOWER]: Advancing to term 2
I20260927 14:59:31.099871 14712 raft_consensus.cc:504] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11b988f649d47b2a387ac1ed39541ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36713 } }
I20260927 14:59:31.100217 14712 leader_election.cc:302] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f11b988f649d47b2a387ac1ed39541ac; no voters:
I20260927 14:59:31.100517 14712 leader_election.cc:288] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 14:59:31.100637 14716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [term 2 FOLLOWER]: Leader election won for term 2
I20260927 14:59:31.100843 14716 raft_consensus.cc:686] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [term 2 LEADER]: Becoming Leader. State: Replica: f11b988f649d47b2a387ac1ed39541ac, State: Running, Role: LEADER
I20260927 14:59:31.101148 14716 consensus_queue.cc:237] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11b988f649d47b2a387ac1ed39541ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36713 } }
I20260927 14:59:31.101823 14712 sys_catalog.cc:564] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:59:31.102171 14716 sys_catalog.cc:455] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader f11b988f649d47b2a387ac1ed39541ac. Latest consensus state: current_term: 2 leader_uuid: "f11b988f649d47b2a387ac1ed39541ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11b988f649d47b2a387ac1ed39541ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36713 } } }
I20260927 14:59:31.102289 14716 sys_catalog.cc:458] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [sys.catalog]: This master's current role is: LEADER
I20260927 14:59:31.102654 14720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:59:31.102674 14717 sys_catalog.cc:455] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f11b988f649d47b2a387ac1ed39541ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11b988f649d47b2a387ac1ed39541ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36713 } } }
I20260927 14:59:31.102937 14717 sys_catalog.cc:458] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [sys.catalog]: This master's current role is: LEADER
I20260927 14:59:31.107990 14720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:59:31.108719 14720 catalog_manager.cc:1259] Loaded cluster ID: 6697c3923601418185e6da925f1a9680
I20260927 14:59:31.108786 14720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:59:31.110291 14720 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 14:59:31.115195 14720 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 14:59:31.121016 14720 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:59:31.121901 14720 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac: Loaded TSK: 0
I20260927 14:59:31.123196 14720 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 14:59:31.181241 14714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:59:31.181546 14714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:59:31.181609 14714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:59:31.185899 14714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:59:31.185987 14714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:59:31.186074 14714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:59:31.193068 14714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40213
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40875
--tserver_master_addrs=127.0.98.62:36713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:59:31.194155 14714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:59:31.195273 14714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:59:31.206427 14739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:31.206434 14740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:31.206434 14742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:31.207345 14714 server_base.cc:1034] running on GCE node
I20260927 14:59:31.207969 14714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:59:31.208706 14714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:59:31.209911 14714 hybrid_clock.cc:648] HybridClock initialized: now 1790521171209884 us; error 46 us; skew 500 ppm
Sep 27 14:59:31 dist-test-slave-pz0h krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:59:31.212898 14714 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:59:31.213910 14714 webserver.cc:460] Webserver started at http://127.0.98.1:40875/ using document root <none> and password file <none>
I20260927 14:59:31.214498 14714 fs_manager.cc:359] Metadata directory not provided
I20260927 14:59:31.214582 14714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:59:31.218786 14714 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:59:31.221132 14749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:31.222667 14714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:59:31.222827 14714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7"
format_stamp: "Formatted at 2026-09-27 14:59:30 on dist-test-slave-pz0h"
server_key: "a134d70a8a0419bf63d9b7f70c41695b"
server_key_iv: "50e7e9921f49a9a6e86abb51d92b283a"
server_key_version: "encryptionkey@0"
I20260927 14:59:31.223438 14714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:59:31.248039 14714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:59:31.251279 14714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:59:31.251461 14714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:59:31.252249 14714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:59:31.254052 14756 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 14:59:31.258539 14714 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 14:59:31.258692 14714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260927 14:59:31.258872 14714 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 14:59:31.260799 14714 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 14:59:31.260859 14714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 14:59:31.261180 14756 tablet_bootstrap.cc:492] T f3bdc605de0146ae96c0365358fc7837 P 648a2752f3ad4e51a06f1ee2db9ff2d7: Bootstrap starting.
I20260927 14:59:31.270820 14714 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40213
I20260927 14:59:31.270939 14863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40213 every 8 connection(s)
I20260927 14:59:31.271845 14714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:59:31.273102 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 14714
I20260927 14:59:31.273571 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37381
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36713
--webserver_port=33465
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:59:31.280222 14756 log.cc:824] T f3bdc605de0146ae96c0365358fc7837 P 648a2752f3ad4e51a06f1ee2db9ff2d7: Log is configured to *not* fsync() on all Append() calls
Sep 27 14:59:31 dist-test-slave-pz0h krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:59:31.289878 14756 tablet_bootstrap.cc:492] T f3bdc605de0146ae96c0365358fc7837 P 648a2752f3ad4e51a06f1ee2db9ff2d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 14:59:31.290184 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:31.273985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38453 (local address 127.0.98.62:36713)
0927 14:59:31.275308 (+ 1323us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:31.275326 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:31.275377 (+ 51us) server_negotiation.cc:407] Connection header received
0927 14:59:31.276646 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:31.276659 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:31.277103 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:31.277450 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:31.279354 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.280936 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.282207 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.282677 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.282880 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:31.285361 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:31.285367 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:31.285380 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:31.285411 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:31.287026 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:31.287548 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:31.287553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:31.287557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:31.287684 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:31.287998 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:31.288000 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:31.288002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:31.288346 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:31.288463 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:31.289038 (+ 575us) server_negotiation.cc:299] Negotiation successful
0927 14:59:31.289304 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1169,"thread_start_us":115,"threads_started":1}
I20260927 14:59:31.290325 14756 tablet_bootstrap.cc:492] T f3bdc605de0146ae96c0365358fc7837 P 648a2752f3ad4e51a06f1ee2db9ff2d7: Bootstrap complete.
I20260927 14:59:31.290594 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:31.273984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.1:38453)
0927 14:59:31.274922 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:31.274990 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:31.276105 (+ 1115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:31.277837 (+ 1732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:31.277852 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:31.278384 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:59:31.279117 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:31.279142 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.281136 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.281140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:31.282063 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:31.282075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.282865 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.282869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:31.283055 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:31.283633 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:59:31.283650 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:59:31.285147 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:59:31.287156 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:31.287162 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:31.287176 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:31.287426 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:31.287773 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:31.287776 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:31.287778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:31.287904 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:31.288454 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:59:31.288460 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:59:31.288711 (+ 251us) client_negotiation.cc:769] Sending connection context
0927 14:59:31.288893 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 14:59:31.289265 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":156,"threads_started":1}
I20260927 14:59:31.291695 14756 ts_tablet_manager.cc:1397] T f3bdc605de0146ae96c0365358fc7837 P 648a2752f3ad4e51a06f1ee2db9ff2d7: Time spent bootstrapping tablet: real 0.031s user 0.020s sys 0.009s
I20260927 14:59:31.292866 14864 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36713
I20260927 14:59:31.293125 14864 heartbeater.cc:464] Registering TS with master...
I20260927 14:59:31.293676 14864 heartbeater.cc:511] Master 127.0.98.62:36713 requested a full tablet report, sending...
I20260927 14:59:31.294942 14756 raft_consensus.cc:348] T f3bdc605de0146ae96c0365358fc7837 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 } } peers { permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 } } peers { permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } }
I20260927 14:59:31.295500 14756 raft_consensus.cc:729] T f3bdc605de0146ae96c0365358fc7837 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 648a2752f3ad4e51a06f1ee2db9ff2d7, State: Initialized, Role: FOLLOWER
I20260927 14:59:31.296010 14756 consensus_queue.cc:260] T f3bdc605de0146ae96c0365358fc7837 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 } } peers { permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 } } peers { permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } }
I20260927 14:59:31.296200 14676 ts_manager.cc:194] Registered new tserver with Master: 648a2752f3ad4e51a06f1ee2db9ff2d7 (127.0.98.1:40213)
I20260927 14:59:31.297106 14756 ts_tablet_manager.cc:1428] T f3bdc605de0146ae96c0365358fc7837 P 648a2752f3ad4e51a06f1ee2db9ff2d7: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260927 14:59:31.299679 14676 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38453
I20260927 14:59:31.302446 14864 heartbeater.cc:503] Master 127.0.98.62:36713 was elected leader, sending a full tablet report...
Sep 27 14:59:31 dist-test-slave-pz0h krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 14:59:31.311117 14871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:31.299221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40213 (local address 127.0.0.1:49728)
0927 14:59:31.300250 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:31.300300 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:31.300457 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:31.301066 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:31.301073 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:31.301516 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:59:31.301663 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:31.301672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.303183 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.303186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:31.304168 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:31.304180 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.304990 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.305010 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:31.305108 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:31.305711 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:59:31.305723 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:59:31.307098 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:59:31.309278 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:31.309289 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:31.309293 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:31.309545 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:31.309894 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:31.309897 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:31.309899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:31.309993 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:31.310512 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:59:31.310518 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:59:31.310632 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 14:59:31.310746 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 14:59:31.310975 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"thread_start_us":141,"threads_started":1}
I20260927 14:59:31.311218 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:31.299293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49728 (local address 127.0.98.1:40213)
0927 14:59:31.299680 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:31.299692 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:31.300606 (+ 914us) server_negotiation.cc:407] Connection header received
0927 14:59:31.300684 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:31.300689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:31.300901 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:31.301040 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:31.301790 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.303036 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.304280 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.304796 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.304903 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:31.307264 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:31.307277 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:31.307291 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:31.307322 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:31.309133 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:31.309661 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:31.309667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:31.309669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:31.309766 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:31.310095 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:31.310098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:31.310100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:31.310400 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:31.310523 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:31.310872 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 14:59:31.311034 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"spinlock_wait_cycles":896,"thread_start_us":86,"threads_started":1}
I20260927 14:59:31.312384 14798 tablet_service.cc:1514] Processing DeleteTablet for tablet f3bdc605de0146ae96c0365358fc7837 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:59:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49728
I20260927 14:59:31.313424 14873 tablet_replica.cc:330] stopping tablet replica
I20260927 14:59:31.313722 14873 raft_consensus.cc:2229] T f3bdc605de0146ae96c0365358fc7837 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:59:31.313961 14873 raft_consensus.cc:2258] T f3bdc605de0146ae96c0365358fc7837 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:59:31.314975 14873 ts_tablet_manager.cc:1905] T f3bdc605de0146ae96c0365358fc7837 P 648a2752f3ad4e51a06f1ee2db9ff2d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:59:31.318667 14873 ts_tablet_manager.cc:1918] T f3bdc605de0146ae96c0365358fc7837 P 648a2752f3ad4e51a06f1ee2db9ff2d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:59:31.318773 14873 log.cc:1196] T f3bdc605de0146ae96c0365358fc7837 P 648a2752f3ad4e51a06f1ee2db9ff2d7: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/wal/wals/f3bdc605de0146ae96c0365358fc7837
I20260927 14:59:31.319010 14873 ts_tablet_manager.cc:1939] T f3bdc605de0146ae96c0365358fc7837 P 648a2752f3ad4e51a06f1ee2db9ff2d7: Deleting consensus metadata
I20260927 14:59:31.321007 14661 catalog_manager.cc:4841] TS 648a2752f3ad4e51a06f1ee2db9ff2d7 (127.0.98.1:40213): tablet f3bdc605de0146ae96c0365358fc7837 (table test-table [id=c40e0db6645547dcb3bbefb57e45c695]) successfully deleted
W20260927 14:59:31.397567 14868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:59:31.397874 14868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:59:31.397919 14868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:59:31.402426 14868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:59:31.402498 14868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:59:31.402562 14868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:59:31.408217 14868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33465
--tserver_master_addrs=127.0.98.62:36713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:59:31.409310 14868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:59:31.410318 14868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:59:31.419919 14878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:31.419981 14879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:31.419947 14881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:31.420223 14868 server_base.cc:1034] running on GCE node
I20260927 14:59:31.420742 14868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:59:31.421396 14868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:59:31.422562 14868 hybrid_clock.cc:648] HybridClock initialized: now 1790521171422547 us; error 29 us; skew 500 ppm
Sep 27 14:59:31 dist-test-slave-pz0h krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:59:31.425392 14868 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:59:31.426322 14868 webserver.cc:460] Webserver started at http://127.0.98.2:33465/ using document root <none> and password file <none>
I20260927 14:59:31.426945 14868 fs_manager.cc:359] Metadata directory not provided
I20260927 14:59:31.427033 14868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:59:31.431645 14868 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:59:31.434926 14888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:31.436406 14868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:59:31.436542 14868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e"
format_stamp: "Formatted at 2026-09-27 14:59:30 on dist-test-slave-pz0h"
server_key: "dbb0fb17cc1a2ae96125889354d0b321"
server_key_iv: "568e28d849702a4dbf32fe404fd013df"
server_key_version: "encryptionkey@0"
I20260927 14:59:31.437150 14868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:59:31.453629 14868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:59:31.457175 14868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:59:31.457360 14868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:59:31.457969 14868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:59:31.459200 14895 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 14:59:31.463403 14868 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 14:59:31.463490 14868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 14:59:31.463551 14868 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 14:59:31.465323 14868 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 14:59:31.465374 14868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 14:59:31.465689 14895 tablet_bootstrap.cc:492] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e: Bootstrap starting.
I20260927 14:59:31.471881 14895 log.cc:824] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e: Log is configured to *not* fsync() on all Append() calls
I20260927 14:59:31.474984 14868 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37381
I20260927 14:59:31.475051 15003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37381 every 8 connection(s)
I20260927 14:59:31.475867 14868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:59:31.481055 14895 tablet_bootstrap.cc:492] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 14:59:31.481449 14895 tablet_bootstrap.cc:492] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e: Bootstrap complete.
I20260927 14:59:31.481797 14895 ts_tablet_manager.cc:1397] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20260927 14:59:31.483273 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 14868
I20260927 14:59:31.483731 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45817
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36713
--webserver_port=43251
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:59:31.484426 14895 raft_consensus.cc:348] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 } } peers { permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 } } peers { permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } }
I20260927 14:59:31.484870 14895 raft_consensus.cc:729] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0f9c4b41077445fa0dc8cc9587f6f0e, State: Initialized, Role: FOLLOWER
I20260927 14:59:31.485723 14895 consensus_queue.cc:260] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 } } peers { permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 } } peers { permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } }
Sep 27 14:59:31 dist-test-slave-pz0h krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:59:31.486745 14895 ts_tablet_manager.cc:1428] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260927 14:59:31.490882 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:31.478051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44909 (local address 127.0.98.62:36713)
0927 14:59:31.478186 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:31.478190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:31.478995 (+ 805us) server_negotiation.cc:407] Connection header received
0927 14:59:31.479979 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:31.479983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:31.480046 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:31.480133 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:31.481326 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.481914 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.483422 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.483816 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.483911 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:31.487300 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:31.487304 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:31.487306 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:31.487338 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:31.488705 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:31.489244 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:31.489247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:31.489248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:31.489331 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:31.489682 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:31.489685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:31.489686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:31.489884 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:31.489973 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:31.490511 (+ 538us) server_negotiation.cc:299] Negotiation successful
0927 14:59:31.490678 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 14:59:31.491719 15006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:31.478233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.2:44909)
0927 14:59:31.478825 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:31.478889 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:31.479765 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:31.480303 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:31.480310 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:31.480678 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:59:31.481175 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:31.481196 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.482057 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.482060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:31.483245 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:31.483260 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.484090 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.484094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:31.484317 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:31.485312 (+ 995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:59:31.485333 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:59:31.487104 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:59:31.488829 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:31.488835 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:31.488852 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:31.489132 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:31.489415 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:31.489418 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:31.489420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:31.489582 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:31.489984 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:59:31.489992 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:59:31.490264 (+ 272us) client_negotiation.cc:769] Sending connection context
0927 14:59:31.490431 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 14:59:31.490828 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":166,"threads_started":1}
I20260927 14:59:31.493327 15004 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36713
I20260927 14:59:31.493574 15004 heartbeater.cc:464] Registering TS with master...
I20260927 14:59:31.494143 15004 heartbeater.cc:511] Master 127.0.98.62:36713 requested a full tablet report, sending...
I20260927 14:59:31.496060 14676 ts_manager.cc:194] Registered new tserver with Master: e0f9c4b41077445fa0dc8cc9587f6f0e (127.0.98.2:37381)
I20260927 14:59:31.497448 14676 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44909
I20260927 14:59:31.500200 15004 heartbeater.cc:503] Master 127.0.98.62:36713 was elected leader, sending a full tablet report...
Sep 27 14:59:31 dist-test-slave-pz0h krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 14:59:31.508409 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:31.497526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41382 (local address 127.0.98.2:37381)
0927 14:59:31.498023 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:31.498035 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:31.498061 (+ 26us) server_negotiation.cc:407] Connection header received
0927 14:59:31.498149 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:31.498155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:31.498406 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:31.499109 (+ 703us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:31.499489 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.500679 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.501564 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.502084 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.502185 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:31.504873 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:31.504886 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:31.504900 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:31.504929 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:31.506436 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:31.507009 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:31.507014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:31.507015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:31.507118 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:31.507433 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:31.507436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:31.507437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:31.507736 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:31.507843 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:31.508126 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 14:59:31.508263 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":127,"threads_started":1}
I20260927 14:59:31.508404 14871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:31.497276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37381 (local address 127.0.0.1:41382)
0927 14:59:31.497647 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:31.497664 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:31.497776 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:31.498568 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:31.498571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:31.498783 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:59:31.498933 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:31.498941 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.500795 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.500799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:31.501438 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:31.501448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.502187 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.502207 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:31.502314 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:31.503049 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:59:31.503062 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:59:31.504669 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:59:31.506567 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:31.506572 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:31.506574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:31.506874 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:31.507211 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:31.507213 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:31.507215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:31.507309 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:31.507834 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:59:31.507837 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:59:31.507950 (+ 113us) client_negotiation.cc:769] Sending connection context
0927 14:59:31.508050 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 14:59:31.508238 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260}
I20260927 14:59:31.509902 14937 tablet_service.cc:1514] Processing DeleteTablet for tablet f3bdc605de0146ae96c0365358fc7837 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:59:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41382
I20260927 14:59:31.511245 15010 tablet_replica.cc:330] stopping tablet replica
I20260927 14:59:31.511585 15010 raft_consensus.cc:2229] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:59:31.511946 15010 raft_consensus.cc:2258] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:59:31.512951 15010 ts_tablet_manager.cc:1905] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:59:31.516928 15010 ts_tablet_manager.cc:1918] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:59:31.517033 15010 log.cc:1196] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/wal/wals/f3bdc605de0146ae96c0365358fc7837
I20260927 14:59:31.517534 15010 ts_tablet_manager.cc:1939] T f3bdc605de0146ae96c0365358fc7837 P e0f9c4b41077445fa0dc8cc9587f6f0e: Deleting consensus metadata
I20260927 14:59:31.520401 14660 catalog_manager.cc:4841] TS e0f9c4b41077445fa0dc8cc9587f6f0e (127.0.98.2:37381): tablet f3bdc605de0146ae96c0365358fc7837 (table test-table [id=c40e0db6645547dcb3bbefb57e45c695]) successfully deleted
W20260927 14:59:31.604230 15007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:59:31.604527 15007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:59:31.604590 15007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:59:31.608785 15007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:59:31.608855 15007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:59:31.608937 15007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:59:31.614502 15007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43251
--tserver_master_addrs=127.0.98.62:36713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:59:31.615669 15007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:59:31.616698 15007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:59:31.626654 15016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:31.626685 15018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:31.626828 15007 server_base.cc:1034] running on GCE node
W20260927 14:59:31.626654 15015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:31.627403 15007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:59:31.628064 15007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:59:31.629248 15007 hybrid_clock.cc:648] HybridClock initialized: now 1790521171629223 us; error 44 us; skew 500 ppm
Sep 27 14:59:31 dist-test-slave-pz0h krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:59:31.631982 15007 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:59:31.632869 15007 webserver.cc:460] Webserver started at http://127.0.98.3:43251/ using document root <none> and password file <none>
I20260927 14:59:31.633409 15007 fs_manager.cc:359] Metadata directory not provided
I20260927 14:59:31.633481 15007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:59:31.637612 15007 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260927 14:59:31.639868 15025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:31.641376 15007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:59:31.641495 15007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "230effef1c7b4942827301a20af55add"
format_stamp: "Formatted at 2026-09-27 14:59:30 on dist-test-slave-pz0h"
server_key: "e6d493fc9f4e8cfd4c476291b0a4b9a9"
server_key_iv: "183b667df6b48df9119f533f5fedcd77"
server_key_version: "encryptionkey@0"
I20260927 14:59:31.642076 15007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:59:31.654485 15007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:59:31.657955 15007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:59:31.658133 15007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:59:31.658718 15007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:59:31.659662 15007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:59:31.659713 15007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:31.659757 15007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:59:31.659778 15007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:31.668331 15007 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45817
I20260927 14:59:31.668376 15138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45817 every 8 connection(s)
I20260927 14:59:31.669407 15007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:59:31.671830 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 15007
Sep 27 14:59:31 dist-test-slave-pz0h krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:59:31.680388 15143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:31.672645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42734 (local address 127.0.98.62:36713)
0927 14:59:31.672839 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:31.672843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:31.672856 (+ 13us) server_negotiation.cc:407] Connection header received
0927 14:59:31.672970 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:31.672974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:31.673026 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:31.673140 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:31.673813 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.674613 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.675528 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.675820 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.675898 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:31.677300 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:31.677304 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:31.677307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:31.677335 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:31.678740 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:31.679196 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:31.679200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:31.679202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:31.679285 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:31.679522 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:31.679525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:31.679527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:31.679741 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:31.679939 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:31.680071 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 14:59:31.680210 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":139,"thread_start_us":58,"threads_started":1}
I20260927 14:59:31.683521 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:31.672293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32815 (local address 127.0.98.62:36713)
0927 14:59:31.672516 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:31.672521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:31.672610 (+ 89us) server_negotiation.cc:407] Connection header received
0927 14:59:31.674106 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:31.674108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:31.674150 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:31.674204 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:31.675562 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.676156 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.677327 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.677537 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.677630 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:31.680194 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:31.680198 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:31.680200 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:31.680226 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:31.681678 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:31.682184 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:31.682188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:31.682190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:31.682267 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:31.682561 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:31.682563 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:31.682564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:31.682715 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:31.682781 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:31.683310 (+ 529us) server_negotiation.cc:299] Negotiation successful
0927 14:59:31.683403 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":147}
I20260927 14:59:31.684365 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:31.671817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.3:32815)
0927 14:59:31.672436 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:31.672497 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:31.673878 (+ 1381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:31.674386 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:31.674395 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:31.674898 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:59:31.675410 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:31.675425 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.676297 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.676301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:31.677192 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:31.677204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.677697 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.677702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:31.677886 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:31.678442 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:59:31.678460 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:59:31.680019 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:59:31.681789 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:31.681794 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:31.681808 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:31.682091 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:31.682350 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:31.682352 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:31.682354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:31.682493 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:31.682832 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:59:31.682839 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:59:31.683090 (+ 251us) client_negotiation.cc:769] Sending connection context
0927 14:59:31.683253 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 14:59:31.683524 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":143,"threads_started":1}
I20260927 14:59:31.685446 15139 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36713
I20260927 14:59:31.685693 15139 heartbeater.cc:464] Registering TS with master...
I20260927 14:59:31.686175 15139 heartbeater.cc:511] Master 127.0.98.62:36713 requested a full tablet report, sending...
I20260927 14:59:31.687278 14676 ts_manager.cc:194] Registered new tserver with Master: 230effef1c7b4942827301a20af55add (127.0.98.3:45817)
I20260927 14:59:31.687836 14676 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:32815
I20260927 14:59:31.694309 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:59:31.699352 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:31.694949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42736 (local address 127.0.98.62:36713)
0927 14:59:31.695078 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:31.695082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:31.695264 (+ 182us) server_negotiation.cc:407] Connection header received
0927 14:59:31.695407 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:31.695409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:31.695522 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:31.695608 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:59:31.695799 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.696314 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.697157 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.697378 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.697458 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:31.697896 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:59:31.698700 (+ 804us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:59:31.698964 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:31.699043 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 14:59:31.699150 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 14:59:31.700203 14676 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:42736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:59:31.704063 14676 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:59:31.711632 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 07072f374b3a4e54987ad893b8da5d54 (DEFAULT_TABLE table=test-table [id=1b4c97d71026481f9395c56e4e2a2fc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:59:31.712176 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07072f374b3a4e54987ad893b8da5d54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:59:31.713596 14798 tablet_service.cc:1467] Processing CreateTablet for tablet 07072f374b3a4e54987ad893b8da5d54 (DEFAULT_TABLE table=test-table [id=1b4c97d71026481f9395c56e4e2a2fc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:59:31.714174 14798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07072f374b3a4e54987ad893b8da5d54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:59:31.714965 14895 tablet_bootstrap.cc:492] T 07072f374b3a4e54987ad893b8da5d54 P e0f9c4b41077445fa0dc8cc9587f6f0e: Bootstrap starting.
I20260927 14:59:31.715919 14895 tablet_bootstrap.cc:654] T 07072f374b3a4e54987ad893b8da5d54 P e0f9c4b41077445fa0dc8cc9587f6f0e: Neither blocks nor log segments found. Creating new log.
I20260927 14:59:31.717048 14756 tablet_bootstrap.cc:492] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7: Bootstrap starting.
I20260927 14:59:31.717872 14756 tablet_bootstrap.cc:654] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7: Neither blocks nor log segments found. Creating new log.
I20260927 14:59:31.718097 14895 tablet_bootstrap.cc:492] T 07072f374b3a4e54987ad893b8da5d54 P e0f9c4b41077445fa0dc8cc9587f6f0e: No bootstrap required, opened a new log
I20260927 14:59:31.718192 14895 ts_tablet_manager.cc:1397] T 07072f374b3a4e54987ad893b8da5d54 P e0f9c4b41077445fa0dc8cc9587f6f0e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 14:59:31.718281 14871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:31.710717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45817 (local address 127.0.0.1:36618)
0927 14:59:31.710907 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:31.710923 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:31.711001 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:31.714081 (+ 3080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:31.714085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:31.714104 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:59:31.714222 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:31.714229 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.715460 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.715463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:31.716657 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:31.716671 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.717771 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.717776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:31.717878 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:31.717929 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 14:59:31.718011 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 14:59:31.718111 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260927 14:59:31.718647 14895 raft_consensus.cc:348] T 07072f374b3a4e54987ad893b8da5d54 P e0f9c4b41077445fa0dc8cc9587f6f0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } } peers { permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 } } peers { permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 } }
I20260927 14:59:31.718755 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:31.710915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.0.98.3:45817)
0927 14:59:31.713592 (+ 2677us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:31.713606 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:31.713633 (+ 27us) server_negotiation.cc:407] Connection header received
0927 14:59:31.713719 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:31.713724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:31.713939 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:31.714087 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:59:31.714332 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.715270 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.716818 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.717614 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.717752 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:31.717911 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:31.718084 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 14:59:31.718294 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2443,"thread_start_us":187,"threads_started":1}
I20260927 14:59:31.718818 14895 raft_consensus.cc:374] T 07072f374b3a4e54987ad893b8da5d54 P e0f9c4b41077445fa0dc8cc9587f6f0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:59:31.718861 14895 raft_consensus.cc:729] T 07072f374b3a4e54987ad893b8da5d54 P e0f9c4b41077445fa0dc8cc9587f6f0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0f9c4b41077445fa0dc8cc9587f6f0e, State: Initialized, Role: FOLLOWER
I20260927 14:59:31.719148 14895 consensus_queue.cc:260] T 07072f374b3a4e54987ad893b8da5d54 P e0f9c4b41077445fa0dc8cc9587f6f0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } } peers { permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 } } peers { permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 } }
I20260927 14:59:31.719393 14756 tablet_bootstrap.cc:492] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7: No bootstrap required, opened a new log
I20260927 14:59:31.719437 14895 ts_tablet_manager.cc:1428] T 07072f374b3a4e54987ad893b8da5d54 P e0f9c4b41077445fa0dc8cc9587f6f0e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 14:59:31.719473 14756 ts_tablet_manager.cc:1397] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 14:59:31.719874 14756 raft_consensus.cc:348] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } } peers { permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 } } peers { permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 } }
I20260927 14:59:31.719998 14756 raft_consensus.cc:374] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:59:31.720048 14756 raft_consensus.cc:729] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 648a2752f3ad4e51a06f1ee2db9ff2d7, State: Initialized, Role: FOLLOWER
I20260927 14:59:31.720173 14756 consensus_queue.cc:260] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } } peers { permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 } } peers { permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 } }
I20260927 14:59:31.720412 14756 ts_tablet_manager.cc:1428] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 14:59:31.722303 15073 tablet_service.cc:1467] Processing CreateTablet for tablet 07072f374b3a4e54987ad893b8da5d54 (DEFAULT_TABLE table=test-table [id=1b4c97d71026481f9395c56e4e2a2fc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:59:31.723428 15073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07072f374b3a4e54987ad893b8da5d54. 1 dirs total, 0 dirs full, 0 dirs failed
W20260927 14:59:31.726835 15005 tablet.cc:2369] T 07072f374b3a4e54987ad893b8da5d54 P e0f9c4b41077445fa0dc8cc9587f6f0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:59:31.730248 15151 tablet_bootstrap.cc:492] T 07072f374b3a4e54987ad893b8da5d54 P 230effef1c7b4942827301a20af55add: Bootstrap starting.
I20260927 14:59:31.732609 15151 tablet_bootstrap.cc:654] T 07072f374b3a4e54987ad893b8da5d54 P 230effef1c7b4942827301a20af55add: Neither blocks nor log segments found. Creating new log.
I20260927 14:59:31.733732 15151 log.cc:824] T 07072f374b3a4e54987ad893b8da5d54 P 230effef1c7b4942827301a20af55add: Log is configured to *not* fsync() on all Append() calls
I20260927 14:59:31.735862 15151 tablet_bootstrap.cc:492] T 07072f374b3a4e54987ad893b8da5d54 P 230effef1c7b4942827301a20af55add: No bootstrap required, opened a new log
I20260927 14:59:31.736079 15151 ts_tablet_manager.cc:1397] T 07072f374b3a4e54987ad893b8da5d54 P 230effef1c7b4942827301a20af55add: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 14:59:31.739383 15151 raft_consensus.cc:348] T 07072f374b3a4e54987ad893b8da5d54 P 230effef1c7b4942827301a20af55add [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } } peers { permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 } } peers { permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 } }
I20260927 14:59:31.739679 15151 raft_consensus.cc:374] T 07072f374b3a4e54987ad893b8da5d54 P 230effef1c7b4942827301a20af55add [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:59:31.739717 15151 raft_consensus.cc:729] T 07072f374b3a4e54987ad893b8da5d54 P 230effef1c7b4942827301a20af55add [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 230effef1c7b4942827301a20af55add, State: Initialized, Role: FOLLOWER
I20260927 14:59:31.740141 15151 consensus_queue.cc:260] T 07072f374b3a4e54987ad893b8da5d54 P 230effef1c7b4942827301a20af55add [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } } peers { permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 } } peers { permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 } }
I20260927 14:59:31.740886 15139 heartbeater.cc:503] Master 127.0.98.62:36713 was elected leader, sending a full tablet report...
I20260927 14:59:31.741212 15151 ts_tablet_manager.cc:1428] T 07072f374b3a4e54987ad893b8da5d54 P 230effef1c7b4942827301a20af55add: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260927 14:59:31.773182 14865 tablet.cc:2369] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:59:31.813424 14870 raft_consensus.cc:482] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:59:31.813655 14870 raft_consensus.cc:504] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } } peers { permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 } } peers { permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 } }
I20260927 14:59:31.814834 14870 leader_election.cc:288] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 230effef1c7b4942827301a20af55add (127.0.98.3:45817), e0f9c4b41077445fa0dc8cc9587f6f0e (127.0.98.2:37381)
I20260927 14:59:31.818766 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:31.815176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53215 (local address 127.0.98.3:45817)
0927 14:59:31.815411 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:31.815415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:31.815608 (+ 193us) server_negotiation.cc:407] Connection header received
0927 14:59:31.815723 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:31.815726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:31.815770 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:31.815830 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:59:31.816084 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.816608 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.817771 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.818185 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.818283 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:31.818349 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:31.818537 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 14:59:31.818632 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260927 14:59:31.818776 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:31.815168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43243 (local address 127.0.98.2:37381)
0927 14:59:31.815299 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:31.815304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:31.815524 (+ 220us) server_negotiation.cc:407] Connection header received
0927 14:59:31.815692 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:31.815694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:31.815746 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:31.815820 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:59:31.816092 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.816696 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.817768 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.818228 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.818323 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:31.818484 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:31.818560 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 14:59:31.818647 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 14:59:31.818841 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:31.815065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37381 (local address 127.0.98.1:43243)
0927 14:59:31.815434 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:31.815447 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:31.815573 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:31.815841 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:31.815844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:31.815861 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:59:31.815995 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:31.816000 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.816793 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.816797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:31.817646 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:31.817656 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.818344 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.818348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:31.818446 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:31.818468 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 14:59:31.818545 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 14:59:31.818652 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":30,"spinlock_wait_cycles":10880,"thread_start_us":102,"threads_started":1}
I20260927 14:59:31.818858 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:31.815076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45817 (local address 127.0.98.1:53215)
0927 14:59:31.815515 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:31.815526 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:31.815625 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:31.815849 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:31.815851 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:31.815880 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:59:31.815985 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:31.815991 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.816710 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.816713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:31.817647 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:31.817657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.818304 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.818308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:31.818401 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:31.818441 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 14:59:31.818510 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 14:59:31.818622 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":216,"spinlock_wait_cycles":2304,"thread_start_us":55,"threads_started":1}
I20260927 14:59:31.819703 15093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07072f374b3a4e54987ad893b8da5d54" candidate_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "230effef1c7b4942827301a20af55add" is_pre_election: true
I20260927 14:59:31.819697 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07072f374b3a4e54987ad893b8da5d54" candidate_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" is_pre_election: true
I20260927 14:59:31.819988 15093 raft_consensus.cc:2454] T 07072f374b3a4e54987ad893b8da5d54 P 230effef1c7b4942827301a20af55add [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 648a2752f3ad4e51a06f1ee2db9ff2d7 in term 0.
I20260927 14:59:31.819988 14957 raft_consensus.cc:2454] T 07072f374b3a4e54987ad893b8da5d54 P e0f9c4b41077445fa0dc8cc9587f6f0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 648a2752f3ad4e51a06f1ee2db9ff2d7 in term 0.
I20260927 14:59:31.820459 14750 leader_election.cc:302] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 648a2752f3ad4e51a06f1ee2db9ff2d7, e0f9c4b41077445fa0dc8cc9587f6f0e; no voters:
I20260927 14:59:31.820756 14870 raft_consensus.cc:2781] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:59:31.820845 14870 raft_consensus.cc:482] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:59:31.820894 14870 raft_consensus.cc:3037] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:59:31.821884 14870 raft_consensus.cc:504] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } } peers { permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 } } peers { permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 } }
I20260927 14:59:31.822219 14870 leader_election.cc:288] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [CANDIDATE]: Term 1 election: Requested vote from peers 230effef1c7b4942827301a20af55add (127.0.98.3:45817), e0f9c4b41077445fa0dc8cc9587f6f0e (127.0.98.2:37381)
I20260927 14:59:31.822630 15093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07072f374b3a4e54987ad893b8da5d54" candidate_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "230effef1c7b4942827301a20af55add"
I20260927 14:59:31.822629 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07072f374b3a4e54987ad893b8da5d54" candidate_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e"
I20260927 14:59:31.822757 14957 raft_consensus.cc:3037] T 07072f374b3a4e54987ad893b8da5d54 P e0f9c4b41077445fa0dc8cc9587f6f0e [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:59:31.822759 15093 raft_consensus.cc:3037] T 07072f374b3a4e54987ad893b8da5d54 P 230effef1c7b4942827301a20af55add [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:59:31.823836 14957 raft_consensus.cc:2454] T 07072f374b3a4e54987ad893b8da5d54 P e0f9c4b41077445fa0dc8cc9587f6f0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 648a2752f3ad4e51a06f1ee2db9ff2d7 in term 1.
I20260927 14:59:31.823859 15093 raft_consensus.cc:2454] T 07072f374b3a4e54987ad893b8da5d54 P 230effef1c7b4942827301a20af55add [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 648a2752f3ad4e51a06f1ee2db9ff2d7 in term 1.
I20260927 14:59:31.824111 14750 leader_election.cc:302] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 648a2752f3ad4e51a06f1ee2db9ff2d7, e0f9c4b41077445fa0dc8cc9587f6f0e; no voters:
I20260927 14:59:31.824288 14870 raft_consensus.cc:2781] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:59:31.824482 14870 raft_consensus.cc:686] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 1 LEADER]: Becoming Leader. State: Replica: 648a2752f3ad4e51a06f1ee2db9ff2d7, State: Running, Role: LEADER
I20260927 14:59:31.824766 14870 consensus_queue.cc:237] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } } peers { permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 } } peers { permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 } }
I20260927 14:59:31.827898 14675 catalog_manager.cc:5495] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 648a2752f3ad4e51a06f1ee2db9ff2d7 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 } health_report { overall_health: UNKNOWN } } }
I20260927 14:59:31.844535 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:31.841096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49740 (local address 127.0.98.1:40213)
0927 14:59:31.841390 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:31.841397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:31.841410 (+ 13us) server_negotiation.cc:407] Connection header received
0927 14:59:31.841487 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:31.841489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:31.841542 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:31.841642 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:59:31.841835 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.842353 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.843182 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:31.843404 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:31.843475 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:31.843681 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:59:31.844157 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:59:31.844242 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:31.844301 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 14:59:31.844356 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":125,"threads_started":1}
I20260927 14:59:31.848507 14957 raft_consensus.cc:1261] T 07072f374b3a4e54987ad893b8da5d54 P e0f9c4b41077445fa0dc8cc9587f6f0e [term 1 FOLLOWER]: Refusing update from remote peer 648a2752f3ad4e51a06f1ee2db9ff2d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:59:31.848511 15093 raft_consensus.cc:1261] T 07072f374b3a4e54987ad893b8da5d54 P 230effef1c7b4942827301a20af55add [term 1 FOLLOWER]: Refusing update from remote peer 648a2752f3ad4e51a06f1ee2db9ff2d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:59:31.849215 14870 consensus_queue.cc:1035] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:59:31.849431 15156 consensus_queue.cc:1035] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:59:31.859048 15160 mvcc.cc:205] Tried to move back new op lower bound from 7333974719884980224 to 7333974719797002240. Current Snapshot: MvccSnapshot[applied={T|T < 7333974719884980224 or (T in {7333974719884980224})}]
I20260927 14:59:31.866004 15161 mvcc.cc:205] Tried to move back new op lower bound from 7333974719884980224 to 7333974719797002240. Current Snapshot: MvccSnapshot[applied={T|T < 7333974719884980224}]
I20260927 14:59:31.867508 15163 mvcc.cc:205] Tried to move back new op lower bound from 7333974719884980224 to 7333974719797002240. Current Snapshot: MvccSnapshot[applied={T|T < 7333974719884980224 or (T in {7333974719884980224})}]
I20260927 14:59:31.871125 14675 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:42736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:59:31.871307 14675 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:42736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:59:31.874441 14675 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac: Sending DeleteTablet for 3 replicas of tablet 07072f374b3a4e54987ad893b8da5d54
I20260927 14:59:31.875491 14798 tablet_service.cc:1514] Processing DeleteTablet for tablet 07072f374b3a4e54987ad893b8da5d54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:59:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49728
I20260927 14:59:31.875583 15073 tablet_service.cc:1514] Processing DeleteTablet for tablet 07072f374b3a4e54987ad893b8da5d54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:59:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36618
I20260927 14:59:31.875897 15169 tablet_replica.cc:330] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7: stopping tablet replica
I20260927 14:59:31.876034 15169 raft_consensus.cc:2229] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:59:31.876346 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 14714
I20260927 14:59:31.876374 15169 raft_consensus.cc:2258] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:59:31.877800 15170 tablet_replica.cc:330] T 07072f374b3a4e54987ad893b8da5d54 P 230effef1c7b4942827301a20af55add: stopping tablet replica
I20260927 14:59:31.878191 15170 raft_consensus.cc:2229] T 07072f374b3a4e54987ad893b8da5d54 P 230effef1c7b4942827301a20af55add [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:59:31.878549 15170 raft_consensus.cc:2258] T 07072f374b3a4e54987ad893b8da5d54 P 230effef1c7b4942827301a20af55add [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:59:31.879060 14937 tablet_service.cc:1514] Processing DeleteTablet for tablet 07072f374b3a4e54987ad893b8da5d54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:59:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41382
I20260927 14:59:31.879261 15010 tablet_replica.cc:330] T 07072f374b3a4e54987ad893b8da5d54 P e0f9c4b41077445fa0dc8cc9587f6f0e: stopping tablet replica
I20260927 14:59:31.879354 15010 raft_consensus.cc:2229] T 07072f374b3a4e54987ad893b8da5d54 P e0f9c4b41077445fa0dc8cc9587f6f0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:59:31.879472 15010 raft_consensus.cc:2258] T 07072f374b3a4e54987ad893b8da5d54 P e0f9c4b41077445fa0dc8cc9587f6f0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:59:31.880095 15170 ts_tablet_manager.cc:1905] T 07072f374b3a4e54987ad893b8da5d54 P 230effef1c7b4942827301a20af55add: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:59:31.880198 15010 ts_tablet_manager.cc:1905] T 07072f374b3a4e54987ad893b8da5d54 P e0f9c4b41077445fa0dc8cc9587f6f0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:59:31.882725 15010 ts_tablet_manager.cc:1918] T 07072f374b3a4e54987ad893b8da5d54 P e0f9c4b41077445fa0dc8cc9587f6f0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:59:31.882727 15170 ts_tablet_manager.cc:1918] T 07072f374b3a4e54987ad893b8da5d54 P 230effef1c7b4942827301a20af55add: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:59:31.882867 15010 log.cc:1196] T 07072f374b3a4e54987ad893b8da5d54 P e0f9c4b41077445fa0dc8cc9587f6f0e: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/wal/wals/07072f374b3a4e54987ad893b8da5d54
I20260927 14:59:31.882880 15170 log.cc:1196] T 07072f374b3a4e54987ad893b8da5d54 P 230effef1c7b4942827301a20af55add: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/wal/wals/07072f374b3a4e54987ad893b8da5d54
I20260927 14:59:31.883116 15010 ts_tablet_manager.cc:1939] T 07072f374b3a4e54987ad893b8da5d54 P e0f9c4b41077445fa0dc8cc9587f6f0e: Deleting consensus metadata
I20260927 14:59:31.883193 15170 ts_tablet_manager.cc:1939] T 07072f374b3a4e54987ad893b8da5d54 P 230effef1c7b4942827301a20af55add: Deleting consensus metadata
I20260927 14:59:31.884024 14660 catalog_manager.cc:4841] TS e0f9c4b41077445fa0dc8cc9587f6f0e (127.0.98.2:37381): tablet 07072f374b3a4e54987ad893b8da5d54 (table test-table [id=1b4c97d71026481f9395c56e4e2a2fc2]) successfully deleted
W20260927 14:59:31.884528 14661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40213 (error 108)
I20260927 14:59:31.885054 14661 catalog_manager.cc:4841] TS 230effef1c7b4942827301a20af55add (127.0.98.3:45817): tablet 07072f374b3a4e54987ad893b8da5d54 (table test-table [id=1b4c97d71026481f9395c56e4e2a2fc2]) successfully deleted
W20260927 14:59:31.885628 14871 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:59:31.885349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40213 (local address 127.0.0.1:49754)
0927 14:59:31.885465 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:31.885480 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:31.885543 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40213: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 14:59:31.885778 14661 catalog_manager.cc:4584] TS 648a2752f3ad4e51a06f1ee2db9ff2d7 (127.0.98.1:40213): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 07072f374b3a4e54987ad893b8da5d54: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40213: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260927 14:59:31.886304 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 14868
I20260927 14:59:31.894606 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 15007
I20260927 14:59:31.901320 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 14643
I20260927 14:59:31.907203 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36713
--webserver_interface=127.0.98.62
--webserver_port=32911
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36713
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:59:32.022459 15172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:59:32.022810 15172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:59:32.022877 15172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:59:32.027479 15172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:59:32.027567 15172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:59:32.027595 15172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:59:32.027618 15172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:59:32.027648 15172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:59:32.033392 15172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=32911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:59:32.034655 15172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:59:32.036051 15172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:59:32.045832 15178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:32.045929 15172 server_base.cc:1034] running on GCE node
W20260927 14:59:32.045832 15177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:32.045845 15180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:32.046739 15172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:59:32.048094 15172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:59:32.049291 15172 hybrid_clock.cc:648] HybridClock initialized: now 1790521172049272 us; error 39 us; skew 500 ppm
Sep 27 14:59:32 dist-test-slave-pz0h krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:59:32.052299 15172 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:59:32.053265 15172 webserver.cc:460] Webserver started at http://127.0.98.62:32911/ using document root <none> and password file <none>
I20260927 14:59:32.053810 15172 fs_manager.cc:359] Metadata directory not provided
I20260927 14:59:32.053870 15172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:59:32.058087 15172 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:59:32.060772 15187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:32.062553 15172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 14:59:32.062731 15172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "f11b988f649d47b2a387ac1ed39541ac"
format_stamp: "Formatted at 2026-09-27 14:59:29 on dist-test-slave-pz0h"
server_key: "e0eb297193e3ec910d1abb0116efa112"
server_key_iv: "4548a50933ae9567a46f8608573dfb77"
server_key_version: "encryptionkey@0"
I20260927 14:59:32.063401 15172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:59:32.080626 15172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:59:32.084022 15172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:59:32.084226 15172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:59:32.091401 15172 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36713
I20260927 14:59:32.091657 15239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36713 every 8 connection(s)
I20260927 14:59:32.092516 15172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:59:32.093931 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 15172
I20260927 14:59:32.094442 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40213
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36713
--webserver_port=40875
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:59:32.097679 15240 sys_catalog.cc:263] Verifying existing consensus state
I20260927 14:59:32.099558 15240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac: Bootstrap starting.
I20260927 14:59:32.117458 15240 log.cc:824] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac: Log is configured to *not* fsync() on all Append() calls
I20260927 14:59:32.128865 15240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 14:59:32.129249 15240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac: Bootstrap complete.
I20260927 14:59:32.133178 15240 raft_consensus.cc:348] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11b988f649d47b2a387ac1ed39541ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36713 } }
I20260927 14:59:32.133581 15240 raft_consensus.cc:729] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f11b988f649d47b2a387ac1ed39541ac, State: Initialized, Role: FOLLOWER
I20260927 14:59:32.134051 15240 consensus_queue.cc:260] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11b988f649d47b2a387ac1ed39541ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36713 } }
I20260927 14:59:32.134166 15240 raft_consensus.cc:388] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:59:32.134229 15240 raft_consensus.cc:482] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:59:32.134320 15240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [term 2 FOLLOWER]: Advancing to term 3
I20260927 14:59:32.136294 15240 raft_consensus.cc:504] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11b988f649d47b2a387ac1ed39541ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36713 } }
I20260927 14:59:32.136603 15240 leader_election.cc:302] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f11b988f649d47b2a387ac1ed39541ac; no voters:
I20260927 14:59:32.136898 15240 leader_election.cc:288] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 14:59:32.137036 15244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [term 3 FOLLOWER]: Leader election won for term 3
I20260927 14:59:32.137277 15244 raft_consensus.cc:686] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [term 3 LEADER]: Becoming Leader. State: Replica: f11b988f649d47b2a387ac1ed39541ac, State: Running, Role: LEADER
I20260927 14:59:32.137593 15244 consensus_queue.cc:237] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11b988f649d47b2a387ac1ed39541ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36713 } }
I20260927 14:59:32.138204 15240 sys_catalog.cc:564] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:59:32.139163 15246 sys_catalog.cc:455] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader f11b988f649d47b2a387ac1ed39541ac. Latest consensus state: current_term: 3 leader_uuid: "f11b988f649d47b2a387ac1ed39541ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11b988f649d47b2a387ac1ed39541ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36713 } } }
I20260927 14:59:32.139303 15246 sys_catalog.cc:458] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [sys.catalog]: This master's current role is: LEADER
I20260927 14:59:32.139586 15245 sys_catalog.cc:455] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f11b988f649d47b2a387ac1ed39541ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11b988f649d47b2a387ac1ed39541ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36713 } } }
I20260927 14:59:32.139665 15245 sys_catalog.cc:458] T 00000000000000000000000000000000 P f11b988f649d47b2a387ac1ed39541ac [sys.catalog]: This master's current role is: LEADER
I20260927 14:59:32.140209 15253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:59:32.145568 15253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:59:32.146298 15253 catalog_manager.cc:1259] Loaded cluster ID: 6697c3923601418185e6da925f1a9680
I20260927 14:59:32.146368 15253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 14:59:32.147459 15253 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1c696d09ad google::LogMessage::Fail() at ??:0
@ 0x7f1c696d388a google::LogMessage::SendToLog() at ??:0
@ 0x7f1c696d04de google::LogMessage::Flush() at ??:0
@ 0x7f1c696d1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1c7400ffdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f1c74010d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1c7400d579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f1c74051640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1c6b9223d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1c6b922c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f1c6b9243f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1c6b913512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1c6cf576db start_thread at ??:0
@ 0x7f1c681ce61f clone at ??:0
W20260927 14:59:32.232312 15242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:59:32.232592 15242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:59:32.232641 15242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:59:32.236881 15242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:59:32.236956 15242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:59:32.237022 15242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:59:32.242599 15242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40213
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40875
--tserver_master_addrs=127.0.98.62:36713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:59:32.243780 15242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:59:32.244865 15242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:59:32.254930 15282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:32.254961 15281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:32.255172 15242 server_base.cc:1034] running on GCE node
W20260927 14:59:32.254930 15284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:32.255726 15242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:59:32.256423 15242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:59:32.257638 15242 hybrid_clock.cc:648] HybridClock initialized: now 1790521172257559 us; error 131 us; skew 500 ppm
Sep 27 14:59:32 dist-test-slave-pz0h krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:59:32.260550 15242 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:59:32.261543 15242 webserver.cc:460] Webserver started at http://127.0.98.1:40875/ using document root <none> and password file <none>
I20260927 14:59:32.262130 15242 fs_manager.cc:359] Metadata directory not provided
I20260927 14:59:32.262202 15242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:59:32.266443 15242 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:59:32.268841 15291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:32.270248 15242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 14:59:32.270388 15242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7"
format_stamp: "Formatted at 2026-09-27 14:59:30 on dist-test-slave-pz0h"
server_key: "a134d70a8a0419bf63d9b7f70c41695b"
server_key_iv: "50e7e9921f49a9a6e86abb51d92b283a"
server_key_version: "encryptionkey@0"
I20260927 14:59:32.271005 15242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:59:32.287017 15242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:59:32.290179 15242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:59:32.290370 15242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:59:32.290992 15242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:59:32.292122 15298 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 14:59:32.297832 15242 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 14:59:32.297938 15242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260927 14:59:32.298018 15242 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 14:59:32.299943 15242 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 14:59:32.300004 15242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20260927 14:59:32.300366 15298 tablet_bootstrap.cc:492] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7: Bootstrap starting.
I20260927 14:59:32.310326 15242 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40213
I20260927 14:59:32.310562 15405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40213 every 8 connection(s)
I20260927 14:59:32.311618 15242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:59:32.311828 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 15242
I20260927 14:59:32.312266 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37381
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36713
--webserver_port=33465
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 14:59:32.321593 15298 log.cc:824] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7: Log is configured to *not* fsync() on all Append() calls
I20260927 14:59:32.331874 15298 tablet_bootstrap.cc:492] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 14:59:32.332338 15298 tablet_bootstrap.cc:492] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7: Bootstrap complete.
I20260927 14:59:32.333721 15298 ts_tablet_manager.cc:1397] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7: Time spent bootstrapping tablet: real 0.034s user 0.021s sys 0.009s
I20260927 14:59:32.337189 15298 raft_consensus.cc:348] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } } peers { permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 } } peers { permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 } }
I20260927 14:59:32.337776 15298 raft_consensus.cc:729] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 648a2752f3ad4e51a06f1ee2db9ff2d7, State: Initialized, Role: FOLLOWER
I20260927 14:59:32.338315 15298 consensus_queue.cc:260] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } } peers { permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 } } peers { permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 } }
I20260927 14:59:32.339424 15298 ts_tablet_manager.cc:1428] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260927 14:59:32.366024 15409 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 14:59:32.314427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.1:44809)
0927 14:59:32.315660 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:32.315727 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:32.316936 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:32.365252 (+ 48316us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":984,"thread_start_us":127,"threads_started":1}
W20260927 14:59:32.366499 15406 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36713 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36713: Client connection negotiation failed: client connection to 127.0.98.62:36713: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 14:59:32.367244 15409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:32.366993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.1:49451)
0927 14:59:32.367114 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:32.367182 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 14:59:32.367830 15409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:32.367635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.1:37201)
0927 14:59:32.367735 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:32.367780 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 14:59:32.367974 15406 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 14:59:32.437757 15408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:59:32.438084 15408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:59:32.438148 15408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:59:32.442451 15408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:59:32.442543 15408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:59:32.442631 15408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:59:32.448443 15408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33465
--tserver_master_addrs=127.0.98.62:36713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:59:32.449613 15408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:59:32.450678 15408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:59:32.461078 15417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:32.461091 15416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:32.461091 15419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:32.461304 15408 server_base.cc:1034] running on GCE node
I20260927 14:59:32.461817 15408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:59:32.462608 15408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:59:32.463801 15408 hybrid_clock.cc:648] HybridClock initialized: now 1790521172463783 us; error 36 us; skew 500 ppm
Sep 27 14:59:32 dist-test-slave-pz0h krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:59:32.466858 15408 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:59:32.467832 15408 webserver.cc:460] Webserver started at http://127.0.98.2:33465/ using document root <none> and password file <none>
I20260927 14:59:32.468410 15408 fs_manager.cc:359] Metadata directory not provided
I20260927 14:59:32.468485 15408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:59:32.472777 15408 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:59:32.475139 15426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:32.477164 15408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 14:59:32.477494 15408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e"
format_stamp: "Formatted at 2026-09-27 14:59:30 on dist-test-slave-pz0h"
server_key: "dbb0fb17cc1a2ae96125889354d0b321"
server_key_iv: "568e28d849702a4dbf32fe404fd013df"
server_key_version: "encryptionkey@0"
I20260927 14:59:32.478214 15408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:59:32.519759 15408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:59:32.523085 15408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:59:32.523272 15408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:59:32.523913 15408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:59:32.524775 15408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:59:32.524824 15408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:32.524879 15408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:59:32.524907 15408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:32.533516 15408 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37381
I20260927 14:59:32.533540 15539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37381 every 8 connection(s)
I20260927 14:59:32.534583 15408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:59:32.538503 15542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:32.536815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.2:49373)
0927 14:59:32.537278 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:32.537769 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":101,"threads_started":1}
W20260927 14:59:32.539057 15540 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36713 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36713: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
I20260927 14:59:32.539650 15542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:32.539444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.2:59969)
0927 14:59:32.539543 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:32.539593 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 14:59:32.540230 15542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:32.540070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.2:58743)
0927 14:59:32.540148 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:32.540187 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 14:59:32.540380 15540 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 14:59:32.544731 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 15408
I20260927 14:59:32.545118 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45817
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36713
--webserver_port=43251
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:59:32.658564 15543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:59:32.658881 15543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:59:32.658928 15543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:59:32.663168 15543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:59:32.663251 15543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:59:32.663316 15543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:59:32.668980 15543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43251
--tserver_master_addrs=127.0.98.62:36713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:59:32.670094 15543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:59:32.671131 15543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:59:32.681165 15551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:32.681159 15549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:32.681159 15548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:32.681638 15543 server_base.cc:1034] running on GCE node
I20260927 14:59:32.682060 15543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:59:32.682859 15543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:59:32.684052 15543 hybrid_clock.cc:648] HybridClock initialized: now 1790521172684027 us; error 40 us; skew 500 ppm
Sep 27 14:59:32 dist-test-slave-pz0h krb5kdc[14096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:59:32.686985 15543 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:59:32.687963 15543 webserver.cc:460] Webserver started at http://127.0.98.3:43251/ using document root <none> and password file <none>
I20260927 14:59:32.688556 15543 fs_manager.cc:359] Metadata directory not provided
I20260927 14:59:32.688644 15543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:59:32.692677 15543 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:59:32.694985 15558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:32.696378 15543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 14:59:32.696512 15543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "230effef1c7b4942827301a20af55add"
format_stamp: "Formatted at 2026-09-27 14:59:30 on dist-test-slave-pz0h"
server_key: "e6d493fc9f4e8cfd4c476291b0a4b9a9"
server_key_iv: "183b667df6b48df9119f533f5fedcd77"
server_key_version: "encryptionkey@0"
I20260927 14:59:32.697122 15543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:59:32.719295 15543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:59:32.722476 15543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:59:32.722674 15543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:59:32.723321 15543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:59:32.724192 15543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:59:32.724243 15543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:32.724298 15543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:59:32.724323 15543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:32.733170 15543 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45817
I20260927 14:59:32.733192 15671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45817 every 8 connection(s)
I20260927 14:59:32.734319 15543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:59:32.738298 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:32.736547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.3:54307)
0927 14:59:32.737022 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:32.737530 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":112,"threads_started":1}
W20260927 14:59:32.738778 15672 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36713 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36713: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
I20260927 14:59:32.739492 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:32.739224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.3:46323)
0927 14:59:32.739372 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:32.739426 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 14:59:32.740144 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:32.739903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.3:52079)
0927 14:59:32.740024 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:32.740076 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 14:59:32.740320 15672 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 14:59:32.741843 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 15543
I20260927 14:59:33.369143 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:33.368652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.1:44147)
0927 14:59:33.368951 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:33.369016 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":77,"threads_started":1}
I20260927 14:59:33.541564 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:33.541059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.2:35387)
0927 14:59:33.541358 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:33.541426 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":93,"threads_started":1}
I20260927 14:59:33.741330 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:33.740924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.3:49369)
0927 14:59:33.741158 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:33.741214 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":72,"threads_started":1}
I20260927 14:59:34.125377 15679 raft_consensus.cc:482] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:59:34.125607 15679 raft_consensus.cc:504] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } } peers { permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 } } peers { permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 } }
I20260927 14:59:34.126711 15679 leader_election.cc:288] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 230effef1c7b4942827301a20af55add (127.0.98.3:45817), e0f9c4b41077445fa0dc8cc9587f6f0e (127.0.98.2:37381)
Sep 27 14:59:34 dist-test-slave-pz0h krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 27 14:59:34 dist-test-slave-pz0h krb5kdc[14096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 14:59:34.141461 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:34.127032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37381 (local address 127.0.98.1:46907)
0927 14:59:34.127373 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:34.127389 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:34.127522 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:34.129155 (+ 1633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:34.129161 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:34.129715 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:59:34.130234 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:34.130252 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:34.131628 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:34.131631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:34.132628 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:34.132638 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:34.133364 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:34.133368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:34.133620 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:34.134283 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:59:34.134306 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:59:34.136338 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:59:34.138899 (+ 2561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:34.138901 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:34.138903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:34.139164 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:34.139648 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:34.139652 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:34.139654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:34.139842 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:34.140527 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:59:34.140529 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:59:34.140674 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 14:59:34.140863 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 14:59:34.141233 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":147,"spinlock_wait_cycles":35712,"thread_start_us":83,"threads_started":1}
I20260927 14:59:34.141482 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:34.127350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35357 (local address 127.0.98.3:45817)
0927 14:59:34.127639 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:34.127660 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:34.127697 (+ 37us) server_negotiation.cc:407] Connection header received
0927 14:59:34.128387 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:34.128393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:34.128755 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:34.129052 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:34.130434 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:34.131476 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:34.132762 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:34.133237 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:34.133392 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:34.136243 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:34.136248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:34.136261 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:34.136293 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:34.138474 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:34.139044 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:34.139051 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:34.139063 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:34.139230 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:34.139648 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:34.139652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:34.139654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:34.140251 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:34.140371 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:34.141033 (+ 662us) server_negotiation.cc:299] Negotiation successful
0927 14:59:34.141319 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":60,"threads_started":1}
I20260927 14:59:34.141515 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:34.127197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46907 (local address 127.0.98.2:37381)
0927 14:59:34.127734 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:34.127745 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:34.127775 (+ 30us) server_negotiation.cc:407] Connection header received
0927 14:59:34.128387 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:34.128393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:34.128754 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:34.129052 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:34.130439 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:34.131476 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:34.132757 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:34.133238 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:34.133392 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:34.136542 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:34.136548 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:34.136562 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:34.136600 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:34.138475 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:34.139314 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:34.139321 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:34.139327 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:34.139505 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:34.139966 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:34.139969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:34.139970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:34.140404 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:34.140542 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:34.141044 (+ 502us) server_negotiation.cc:299] Negotiation successful
0927 14:59:34.141321 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":72,"threads_started":1}
I20260927 14:59:34.141500 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:34.127038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45817 (local address 127.0.98.1:35357)
0927 14:59:34.127417 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:34.127431 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:34.127539 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:34.129155 (+ 1616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:34.129161 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:34.129715 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:59:34.130235 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:34.130253 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:34.131632 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:34.131635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:34.132628 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:34.132638 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:34.133364 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:34.133368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:34.133598 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:34.134333 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:59:34.134348 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:59:34.135978 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:59:34.138613 (+ 2635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:34.138618 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:34.138630 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:34.138923 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:34.139356 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:34.139359 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:34.139361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:34.139522 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:34.140403 (+ 881us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:59:34.140410 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:59:34.140752 (+ 342us) client_negotiation.cc:769] Sending connection context
0927 14:59:34.140877 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 14:59:34.141321 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":351,"spinlock_wait_cycles":1280,"thread_start_us":76,"threads_started":1}
W20260927 14:59:34.144104 15292 leader_election.cc:341] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0f9c4b41077445fa0dc8cc9587f6f0e (127.0.98.2:37381): Not found: Tablet not found: 07072f374b3a4e54987ad893b8da5d54
W20260927 14:59:34.144259 15293 leader_election.cc:341] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 230effef1c7b4942827301a20af55add (127.0.98.3:45817): Not found: Tablet not found: 07072f374b3a4e54987ad893b8da5d54
I20260927 14:59:34.144374 15293 leader_election.cc:302] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 648a2752f3ad4e51a06f1ee2db9ff2d7; no voters: 230effef1c7b4942827301a20af55add, e0f9c4b41077445fa0dc8cc9587f6f0e
I20260927 14:59:34.144673 15679 raft_consensus.cc:2726] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 14:59:34.370260 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:34.369968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.1:55269)
0927 14:59:34.370116 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:34.370172 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 14:59:34.542711 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:34.542307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.2:42053)
0927 14:59:34.542531 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:34.542580 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":70,"threads_started":1}
I20260927 14:59:34.742480 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:34.742098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.3:35741)
0927 14:59:34.742334 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:34.742387 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":77,"threads_started":1}
I20260927 14:59:35.371621 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:35.371169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.1:40625)
0927 14:59:35.371445 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:35.371513 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":82,"threads_started":1}
I20260927 14:59:35.544021 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:35.543541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.2:41707)
0927 14:59:35.543834 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:35.543906 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":105,"threads_started":1}
I20260927 14:59:35.743801 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:35.743334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.3:54045)
0927 14:59:35.743621 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:35.743683 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":92,"threads_started":1}
I20260927 14:59:35.938344 15689 raft_consensus.cc:482] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:59:35.938496 15689 raft_consensus.cc:504] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } } peers { permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 } } peers { permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 } }
I20260927 14:59:35.938982 15689 leader_election.cc:288] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 230effef1c7b4942827301a20af55add (127.0.98.3:45817), e0f9c4b41077445fa0dc8cc9587f6f0e (127.0.98.2:37381)
W20260927 14:59:35.939816 15293 leader_election.cc:341] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 230effef1c7b4942827301a20af55add (127.0.98.3:45817): Not found: Tablet not found: 07072f374b3a4e54987ad893b8da5d54
W20260927 14:59:35.939942 15292 leader_election.cc:341] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0f9c4b41077445fa0dc8cc9587f6f0e (127.0.98.2:37381): Not found: Tablet not found: 07072f374b3a4e54987ad893b8da5d54
I20260927 14:59:35.940028 15292 leader_election.cc:302] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 648a2752f3ad4e51a06f1ee2db9ff2d7; no voters: 230effef1c7b4942827301a20af55add, e0f9c4b41077445fa0dc8cc9587f6f0e
I20260927 14:59:35.940201 15689 raft_consensus.cc:2726] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 14:59:36.373133 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:36.372631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.1:49265)
0927 14:59:36.372948 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:36.373012 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":102,"threads_started":1}
I20260927 14:59:36.545301 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:36.544829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.2:37759)
0927 14:59:36.545130 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:36.545196 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":81,"threads_started":1}
I20260927 14:59:36.745289 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:36.744813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.3:42251)
0927 14:59:36.745104 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:36.745183 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":86,"threads_started":1}
I20260927 14:59:37.375241 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:37.374389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.1:37443)
0927 14:59:37.375007 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:37.375094 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":260,"threads_started":1}
I20260927 14:59:37.546823 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:37.546288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.2:34335)
0927 14:59:37.546615 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:37.546690 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":126,"threads_started":1}
I20260927 14:59:37.746831 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:37.746299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.3:48757)
0927 14:59:37.746610 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:37.746690 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":83,"threads_started":1}
I20260927 14:59:38.376477 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:38.376028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.1:43893)
0927 14:59:38.376297 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:38.376366 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":86,"threads_started":1}
I20260927 14:59:38.547988 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:38.547524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.2:60015)
0927 14:59:38.547810 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:38.547880 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":80,"threads_started":1}
I20260927 14:59:38.748152 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:38.747725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.3:58747)
0927 14:59:38.747993 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:38.748054 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":77,"threads_started":1}
I20260927 14:59:38.899689 15699 raft_consensus.cc:482] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:59:38.899830 15699 raft_consensus.cc:504] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } } peers { permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 } } peers { permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 } }
I20260927 14:59:38.900238 15699 leader_election.cc:288] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 230effef1c7b4942827301a20af55add (127.0.98.3:45817), e0f9c4b41077445fa0dc8cc9587f6f0e (127.0.98.2:37381)
W20260927 14:59:38.901060 15293 leader_election.cc:341] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 230effef1c7b4942827301a20af55add (127.0.98.3:45817): Not found: Tablet not found: 07072f374b3a4e54987ad893b8da5d54
W20260927 14:59:38.901185 15292 leader_election.cc:341] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0f9c4b41077445fa0dc8cc9587f6f0e (127.0.98.2:37381): Not found: Tablet not found: 07072f374b3a4e54987ad893b8da5d54
I20260927 14:59:38.901284 15292 leader_election.cc:302] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 648a2752f3ad4e51a06f1ee2db9ff2d7; no voters: 230effef1c7b4942827301a20af55add, e0f9c4b41077445fa0dc8cc9587f6f0e
I20260927 14:59:38.901427 15699 raft_consensus.cc:2726] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 14:59:39.377669 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:39.377243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.1:42339)
0927 14:59:39.377507 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:39.377565 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":76,"threads_started":1}
I20260927 14:59:39.549351 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:39.548838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.2:46169)
0927 14:59:39.549168 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:39.549237 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":75,"threads_started":1}
I20260927 14:59:39.749761 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:39.749051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.3:47009)
0927 14:59:39.749566 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:39.749641 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":94,"threads_started":1}
I20260927 14:59:40.379036 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:40.378545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.1:41331)
0927 14:59:40.378867 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:40.378933 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":94,"threads_started":1}
I20260927 14:59:40.550631 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:40.550169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.2:57261)
0927 14:59:40.550464 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:40.550529 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":80,"threads_started":1}
I20260927 14:59:40.751715 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:40.750992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.3:60715)
0927 14:59:40.751426 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:40.751568 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":174,"threads_started":1}
I20260927 14:59:41.380261 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:41.379797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.1:59483)
0927 14:59:41.380077 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:41.380146 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":96,"threads_started":1}
I20260927 14:59:41.551990 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:41.551521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.2:48057)
0927 14:59:41.551819 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:41.551883 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":79,"threads_started":1}
I20260927 14:59:41.753320 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:41.752768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.3:55711)
0927 14:59:41.753089 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:41.753169 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":118,"threads_started":1}
I20260927 14:59:42.381842 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:42.381278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.1:58471)
0927 14:59:42.381607 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:42.381686 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":101,"threads_started":1}
I20260927 14:59:42.553386 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:42.552946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.2:47507)
0927 14:59:42.553219 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:42.553285 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":92,"threads_started":1}
I20260927 14:59:42.754760 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:42.754289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.3:47913)
0927 14:59:42.754573 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:42.754642 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":92,"threads_started":1}
I20260927 14:59:43.383172 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:43.382612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.1:53271)
0927 14:59:43.382992 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:43.383057 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":113,"threads_started":1}
I20260927 14:59:43.476871 15713 raft_consensus.cc:482] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:59:43.477041 15713 raft_consensus.cc:504] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230effef1c7b4942827301a20af55add" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } } peers { permanent_uuid: "648a2752f3ad4e51a06f1ee2db9ff2d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40213 } } peers { permanent_uuid: "e0f9c4b41077445fa0dc8cc9587f6f0e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37381 } }
I20260927 14:59:43.477488 15713 leader_election.cc:288] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 230effef1c7b4942827301a20af55add (127.0.98.3:45817), e0f9c4b41077445fa0dc8cc9587f6f0e (127.0.98.2:37381)
W20260927 14:59:43.478251 15292 leader_election.cc:341] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0f9c4b41077445fa0dc8cc9587f6f0e (127.0.98.2:37381): Not found: Tablet not found: 07072f374b3a4e54987ad893b8da5d54
W20260927 14:59:43.478368 15293 leader_election.cc:341] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 230effef1c7b4942827301a20af55add (127.0.98.3:45817): Not found: Tablet not found: 07072f374b3a4e54987ad893b8da5d54
I20260927 14:59:43.478457 15293 leader_election.cc:302] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 648a2752f3ad4e51a06f1ee2db9ff2d7; no voters: 230effef1c7b4942827301a20af55add, e0f9c4b41077445fa0dc8cc9587f6f0e
I20260927 14:59:43.478617 15713 raft_consensus.cc:2726] T 07072f374b3a4e54987ad893b8da5d54 P 648a2752f3ad4e51a06f1ee2db9ff2d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 14:59:43.554641 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:43.554178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.2:41591)
0927 14:59:43.554484 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:43.554544 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":95,"threads_started":1}
I20260927 14:59:43.756337 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:43.755642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.3:41161)
0927 14:59:43.756141 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:43.756227 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":92,"threads_started":1}
I20260927 14:59:44.384574 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:44.384063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.1:52501)
0927 14:59:44.384390 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:44.384467 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":107,"threads_started":1}
I20260927 14:59:44.555972 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:44.555520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.2:43235)
0927 14:59:44.555803 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:44.555870 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":78,"threads_started":1}
I20260927 14:59:44.757741 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:44.757243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.3:48619)
0927 14:59:44.757545 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:44.757613 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":97,"threads_started":1}
I20260927 14:59:45.385901 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:45.385356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.1:45657)
0927 14:59:45.385650 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:45.385796 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":75,"threads_started":1}
I20260927 14:59:45.557224 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:45.556759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.2:34813)
0927 14:59:45.557047 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:45.557115 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":87,"threads_started":1}
I20260927 14:59:45.759105 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:45.758617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.3:58027)
0927 14:59:45.758939 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:45.759002 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":84,"threads_started":1}
I20260927 14:59:46.387192 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:46.386753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.1:47763)
0927 14:59:46.387032 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:46.387094 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":119,"threads_started":1}
I20260927 14:59:46.558610 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:46.558120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.2:58357)
0927 14:59:46.558441 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:46.558508 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":98,"threads_started":1}
I20260927 14:59:46.760449 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:46.759959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.3:50687)
0927 14:59:46.760267 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:46.760333 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":75,"threads_started":1}
I20260927 14:59:47.389381 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:47.388714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.1:58055)
0927 14:59:47.389168 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:47.389257 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":111,"threads_started":1}
I20260927 14:59:47.560066 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:47.559545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36713 (local address 127.0.98.2:55719)
0927 14:59:47.559880 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:47.559959 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":78,"threads_started":1}
I20260927 14:59:47.747159 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 15242
I20260927 14:59:47.752580 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 15408
I20260927 14:59:47.757875 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 15543
2026-09-27T14:59:47Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19256 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 14:59:47 dist-test-slave-pz0h krb5kdc[15729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 14:59:47 dist-test-slave-pz0h krb5kdc[15729](info): set up 2 sockets
Sep 27 14:59:47 dist-test-slave-pz0h krb5kdc[15729](info): commencing operation
krb5kdc: starting...
W20260927 14:59:48.726917 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 14:59:48 dist-test-slave-pz0h krb5kdc[15729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T14:59:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T14:59:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 14:59:48.849999 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34767
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:59:48.968955 15742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:59:48.969285 15742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:59:48.969345 15742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:59:48.973562 15742 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 14:59:48.973641 15742 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 14:59:48.973661 15742 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 14:59:48.973678 15742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 14:59:48.973703 15742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:59:48.973727 15742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:59:48.973749 15742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 14:59:48.973778 15742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 14:59:48.979547 15742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:59:48.980738 15742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:59:48.981792 15742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:59:48.990536 15748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:48.990589 15742 server_base.cc:1034] running on GCE node
W20260927 14:59:48.990538 15747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:48.990536 15750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:48.991466 15742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:59:48.992667 15742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:59:48.993870 15742 hybrid_clock.cc:648] HybridClock initialized: now 1790521188993854 us; error 33 us; skew 500 ppm
Sep 27 14:59:48 dist-test-slave-pz0h krb5kdc[15729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:59:48.996851 15742 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 14:59:48.997860 15742 webserver.cc:460] Webserver started at http://127.0.98.62:39615/ using document root <none> and password file <none>
I20260927 14:59:48.998433 15742 fs_manager.cc:359] Metadata directory not provided
I20260927 14:59:48.998503 15742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:59:48.998714 15742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:59:49.000923 15742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "2b08eee532aa45d88003af67633d057a"
format_stamp: "Formatted at 2026-09-27 14:59:48 on dist-test-slave-pz0h"
server_key: "99e5367e5fedfb1a4eeabaab4fb95e20"
server_key_iv: "7e3be957b1ce1e1cbf40e419ba2cd936"
server_key_version: "encryptionkey@0"
I20260927 14:59:49.001516 15742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "2b08eee532aa45d88003af67633d057a"
format_stamp: "Formatted at 2026-09-27 14:59:48 on dist-test-slave-pz0h"
server_key: "99e5367e5fedfb1a4eeabaab4fb95e20"
server_key_iv: "7e3be957b1ce1e1cbf40e419ba2cd936"
server_key_version: "encryptionkey@0"
I20260927 14:59:49.006064 15742 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 14:59:49.009547 15757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:49.011076 15742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:59:49.011230 15742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "2b08eee532aa45d88003af67633d057a"
format_stamp: "Formatted at 2026-09-27 14:59:48 on dist-test-slave-pz0h"
server_key: "99e5367e5fedfb1a4eeabaab4fb95e20"
server_key_iv: "7e3be957b1ce1e1cbf40e419ba2cd936"
server_key_version: "encryptionkey@0"
I20260927 14:59:49.011365 15742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:59:49.024271 15742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:59:49.027838 15742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:59:49.028038 15742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:59:49.035197 15742 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34767
I20260927 14:59:49.035485 15809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34767 every 8 connection(s)
I20260927 14:59:49.036228 15742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 14:59:49.037161 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 15742
I20260927 14:59:49.037281 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 14:59:49.037578 392 external_mini_cluster.cc:1428] Setting key b3cf1c5475c7d13064c090816593740a
I20260927 14:59:49.039637 15810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:59:49.046821 15810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a: Bootstrap starting.
Sep 27 14:59:49 dist-test-slave-pz0h krb5kdc[15729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:59:49.049651 15810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a: Neither blocks nor log segments found. Creating new log.
I20260927 14:59:49.050444 15810 log.cc:824] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a: Log is configured to *not* fsync() on all Append() calls
I20260927 14:59:49.052538 15810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a: No bootstrap required, opened a new log
I20260927 14:59:49.052618 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:49.039049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57612 (local address 127.0.98.62:34767)
0927 14:59:49.039497 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:49.039513 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:49.039551 (+ 38us) server_negotiation.cc:407] Connection header received
0927 14:59:49.040247 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:49.040266 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:49.040617 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:49.040941 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:49.041895 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:49.043024 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:49.044206 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:49.044734 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:49.044875 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:49.047919 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:49.047930 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:49.047946 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:49.047985 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:49.049684 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:49.050180 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:49.050186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:49.050192 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:49.050315 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:49.050580 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:49.050584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:49.050586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:49.051065 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:49.051260 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:49.051552 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 14:59:49.051858 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":140,"threads_started":1}
I20260927 14:59:49.056713 15810 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b08eee532aa45d88003af67633d057a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } }
I20260927 14:59:49.056998 15810 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:59:49.057062 15810 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b08eee532aa45d88003af67633d057a, State: Initialized, Role: FOLLOWER
I20260927 14:59:49.057509 15810 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b08eee532aa45d88003af67633d057a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } }
I20260927 14:59:49.057657 15810 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 14:59:49.057734 15810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 14:59:49.057834 15810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:59:49.058912 15810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b08eee532aa45d88003af67633d057a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } }
I20260927 14:59:49.059326 15810 leader_election.cc:302] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b08eee532aa45d88003af67633d057a; no voters:
I20260927 14:59:49.059705 15810 leader_election.cc:288] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 14:59:49.059862 15815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:59:49.060452 15815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [term 1 LEADER]: Becoming Leader. State: Replica: 2b08eee532aa45d88003af67633d057a, State: Running, Role: LEADER
I20260927 14:59:49.061137 15810 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 14:59:49.061163 15815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b08eee532aa45d88003af67633d057a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } }
I20260927 14:59:49.063120 15816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b08eee532aa45d88003af67633d057a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b08eee532aa45d88003af67633d057a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } } }
I20260927 14:59:49.063472 15816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [sys.catalog]: This master's current role is: LEADER
I20260927 14:59:49.063081 15817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b08eee532aa45d88003af67633d057a. Latest consensus state: current_term: 1 leader_uuid: "2b08eee532aa45d88003af67633d057a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b08eee532aa45d88003af67633d057a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } } }
I20260927 14:59:49.063807 15817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [sys.catalog]: This master's current role is: LEADER
W20260927 14:59:49.066246 15829 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 14:59:49.066355 15829 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 14:59:49.066459 15830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 14:59:49.067317 15830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 14:59:49.073945 15830 catalog_manager.cc:1347] Generated new cluster ID: e0a207cee683435db0e36621f7e4cac1
I20260927 14:59:49.074028 15830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 14:59:49.079236 15830 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 14:59:49.080259 15830 catalog_manager.cc:1504] Loading token signing keys...
I20260927 14:59:49.090042 15830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a: Generated new TSK 0
I20260927 14:59:49.090865 15830 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 14:59:49.128283 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34767
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:59:49.268963 15836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:59:49.269287 15836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:59:49.269351 15836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:59:49.273612 15836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:59:49.273705 15836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:59:49.273787 15836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 14:59:49.279482 15836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:59:49.280797 15836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:59:49.281783 15836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:59:49.291754 15842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:49.292007 15841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:49.292007 15836 server_base.cc:1034] running on GCE node
W20260927 14:59:49.292284 15844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:49.292912 15836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:59:49.293646 15836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:59:49.294863 15836 hybrid_clock.cc:648] HybridClock initialized: now 1790521189294846 us; error 30 us; skew 500 ppm
Sep 27 14:59:49 dist-test-slave-pz0h krb5kdc[15729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:59:49.298264 15836 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 14:59:49.299439 15836 webserver.cc:460] Webserver started at http://127.0.98.1:37049/ using document root <none> and password file <none>
I20260927 14:59:49.300014 15836 fs_manager.cc:359] Metadata directory not provided
I20260927 14:59:49.300092 15836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:59:49.300312 15836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:59:49.302477 15836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/data/instance:
uuid: "2e934c9fbd86496fa240f8cbfce7c05b"
format_stamp: "Formatted at 2026-09-27 14:59:49 on dist-test-slave-pz0h"
server_key: "4473e166550f139ffcc405c8b25b5b8c"
server_key_iv: "b4412227f296a39a1110fe733686bf39"
server_key_version: "encryptionkey@0"
I20260927 14:59:49.303143 15836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2e934c9fbd86496fa240f8cbfce7c05b"
format_stamp: "Formatted at 2026-09-27 14:59:49 on dist-test-slave-pz0h"
server_key: "4473e166550f139ffcc405c8b25b5b8c"
server_key_iv: "b4412227f296a39a1110fe733686bf39"
server_key_version: "encryptionkey@0"
I20260927 14:59:49.307539 15836 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 14:59:49.310717 15851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:49.312328 15836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 14:59:49.312498 15836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "2e934c9fbd86496fa240f8cbfce7c05b"
format_stamp: "Formatted at 2026-09-27 14:59:49 on dist-test-slave-pz0h"
server_key: "4473e166550f139ffcc405c8b25b5b8c"
server_key_iv: "b4412227f296a39a1110fe733686bf39"
server_key_version: "encryptionkey@0"
I20260927 14:59:49.312646 15836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:59:49.335143 15836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:59:49.338445 15836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:59:49.338658 15836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:59:49.339322 15836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:59:49.340212 15836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:59:49.340262 15836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:49.340317 15836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:59:49.340343 15836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:49.349529 15836 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46197
I20260927 14:59:49.349555 15964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46197 every 8 connection(s)
I20260927 14:59:49.350551 15836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 14:59:49.356613 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 15836
I20260927 14:59:49.356735 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/wal/instance
I20260927 14:59:49.356982 392 external_mini_cluster.cc:1428] Setting key 6e59cb4c7f2539b5d6ee2fe2987171a6
Sep 27 14:59:49 dist-test-slave-pz0h krb5kdc[15729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:59:49.366371 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:49.353038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40923 (local address 127.0.98.62:34767)
0927 14:59:49.353230 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:49.353234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:49.354256 (+ 1022us) server_negotiation.cc:407] Connection header received
0927 14:59:49.355642 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:49.355645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:49.355704 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:49.355778 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:49.357071 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:49.357620 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:49.358826 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:49.359087 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:49.359182 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:49.362326 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:49.362330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:49.362334 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:49.362369 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:49.364012 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:49.364643 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:49.364648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:49.364649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:49.364754 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:49.365080 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:49.365083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:49.365085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:49.365339 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:49.365446 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:49.366026 (+ 580us) server_negotiation.cc:299] Negotiation successful
0927 14:59:49.366189 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 14:59:49.367226 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:49.353499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:40923)
0927 14:59:49.354052 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:49.354120 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:49.355411 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:49.355968 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:49.355976 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:49.356374 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:59:49.356914 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:49.356933 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:49.357748 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:49.357751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:49.358674 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:49.358685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:49.359240 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:49.359245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:49.359429 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:49.360110 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:59:49.360132 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:59:49.362094 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:59:49.364187 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:49.364193 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:49.364209 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:49.364521 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:49.364846 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:49.364848 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:49.364850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:49.364988 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:49.365459 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:59:49.365465 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:59:49.365767 (+ 302us) client_negotiation.cc:769] Sending connection context
0927 14:59:49.365943 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 14:59:49.366242 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":157,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 14:59:49.368649 15965 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34767
I20260927 14:59:49.368901 15965 heartbeater.cc:464] Registering TS with master...
I20260927 14:59:49.369505 15965 heartbeater.cc:511] Master 127.0.98.62:34767 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 14:59:49.371443 15774 ts_manager.cc:194] Registered new tserver with Master: 2e934c9fbd86496fa240f8cbfce7c05b (127.0.98.1:46197)
I20260927 14:59:49.372695 15774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 14:59:49.386817 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34767
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:59:49.503819 15970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:59:49.504109 15970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:59:49.504154 15970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:59:49.508345 15970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:59:49.508424 15970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:59:49.508486 15970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 14:59:49.513978 15970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:59:49.515161 15970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:59:49.516317 15970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:59:49.526393 15976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:49.526503 15978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:49.526535 15975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:49.526719 15970 server_base.cc:1034] running on GCE node
I20260927 14:59:49.527194 15970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:59:49.527868 15970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:59:49.529059 15970 hybrid_clock.cc:648] HybridClock initialized: now 1790521189529028 us; error 37 us; skew 500 ppm
Sep 27 14:59:49 dist-test-slave-pz0h krb5kdc[15729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:59:49.532044 15970 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 14:59:49.533090 15970 webserver.cc:460] Webserver started at http://127.0.98.2:33521/ using document root <none> and password file <none>
I20260927 14:59:49.533679 15970 fs_manager.cc:359] Metadata directory not provided
I20260927 14:59:49.533751 15970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:59:49.533970 15970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:59:49.536316 15970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/data/instance:
uuid: "4b093d925204400bbc7da11cd8629238"
format_stamp: "Formatted at 2026-09-27 14:59:49 on dist-test-slave-pz0h"
server_key: "256fd6339acfeba332ca4f967668b99f"
server_key_iv: "8e0ba0bdc18e5bc67593adf95b6f54b4"
server_key_version: "encryptionkey@0"
I20260927 14:59:49.536944 15970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4b093d925204400bbc7da11cd8629238"
format_stamp: "Formatted at 2026-09-27 14:59:49 on dist-test-slave-pz0h"
server_key: "256fd6339acfeba332ca4f967668b99f"
server_key_iv: "8e0ba0bdc18e5bc67593adf95b6f54b4"
server_key_version: "encryptionkey@0"
I20260927 14:59:49.541594 15970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 14:59:49.544853 15985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:49.546213 15970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260927 14:59:49.546361 15970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "4b093d925204400bbc7da11cd8629238"
format_stamp: "Formatted at 2026-09-27 14:59:49 on dist-test-slave-pz0h"
server_key: "256fd6339acfeba332ca4f967668b99f"
server_key_iv: "8e0ba0bdc18e5bc67593adf95b6f54b4"
server_key_version: "encryptionkey@0"
I20260927 14:59:49.546489 15970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:59:49.563555 15970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:59:49.566488 15970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:59:49.566669 15970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:59:49.567309 15970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:59:49.568161 15970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:59:49.568209 15970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:49.568264 15970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:59:49.568287 15970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:49.576697 15970 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33955
I20260927 14:59:49.576726 16098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33955 every 8 connection(s)
I20260927 14:59:49.577629 15970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 14:59:49.583688 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 15970
I20260927 14:59:49.583810 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/wal/instance
I20260927 14:59:49.584057 392 external_mini_cluster.cc:1428] Setting key 0f45fc19b0e5c18918e065bc5c4293b5
Sep 27 14:59:49 dist-test-slave-pz0h krb5kdc[15729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:59:49.591918 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:49.579431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35353 (local address 127.0.98.62:34767)
0927 14:59:49.579581 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:49.579586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:49.580395 (+ 809us) server_negotiation.cc:407] Connection header received
0927 14:59:49.581669 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:49.581673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:49.581730 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:49.581802 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:49.583073 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:49.583691 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:49.585065 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:49.585324 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:49.585409 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:49.587984 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:49.587988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:49.588002 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:49.588031 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:49.589473 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:49.590161 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:49.590165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:49.590167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:49.590268 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:49.590680 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:49.590682 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:49.590684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:49.590903 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:49.590979 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:49.591654 (+ 675us) server_negotiation.cc:299] Negotiation successful
0927 14:59:49.591794 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 14:59:49.593175 16101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:49.579787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:35353)
0927 14:59:49.580245 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:49.580289 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:49.581459 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:49.581975 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:49.581983 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:49.582352 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:59:49.582909 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:49.582928 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:49.583869 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:49.583873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:49.584940 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:49.584953 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:49.585481 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:49.585485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:49.585647 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:49.586243 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:59:49.586262 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:59:49.587761 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:59:49.589615 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:49.589623 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:49.589643 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:49.590048 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:49.590362 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:49.590365 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:49.590367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:49.590593 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:49.590997 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:59:49.591005 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:59:49.591383 (+ 378us) client_negotiation.cc:769] Sending connection context
0927 14:59:49.591629 (+ 246us) client_negotiation.cc:240] Negotiation successful
0927 14:59:49.592027 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 14:59:49.594307 16099 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34767
I20260927 14:59:49.594559 16099 heartbeater.cc:464] Registering TS with master...
I20260927 14:59:49.595083 16099 heartbeater.cc:511] Master 127.0.98.62:34767 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 14:59:49.596174 15774 ts_manager.cc:194] Registered new tserver with Master: 4b093d925204400bbc7da11cd8629238 (127.0.98.2:33955)
I20260927 14:59:49.596758 15774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 14:59:49.609876 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34767
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 14:59:49.734148 16104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 14:59:49.734438 16104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 14:59:49.734503 16104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 14:59:49.738984 16104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 14:59:49.739058 16104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 14:59:49.739148 16104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 14:59:49.744735 16104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 14:59:49.746006 16104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 14:59:49.747150 16104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 14:59:49.756860 16109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:49.756860 16110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 14:59:49.756860 16112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 14:59:49.757121 16104 server_base.cc:1034] running on GCE node
I20260927 14:59:49.757529 16104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 14:59:49.758183 16104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 14:59:49.759352 16104 hybrid_clock.cc:648] HybridClock initialized: now 1790521189759331 us; error 31 us; skew 500 ppm
Sep 27 14:59:49 dist-test-slave-pz0h krb5kdc[15729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 14:59:49.762531 16104 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 14:59:49.763581 16104 webserver.cc:460] Webserver started at http://127.0.98.3:44667/ using document root <none> and password file <none>
I20260927 14:59:49.764146 16104 fs_manager.cc:359] Metadata directory not provided
I20260927 14:59:49.764214 16104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 14:59:49.764405 16104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 14:59:49.766480 16104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/data/instance:
uuid: "52a91d287a1e40c58f1f7f7951c20648"
format_stamp: "Formatted at 2026-09-27 14:59:49 on dist-test-slave-pz0h"
server_key: "50583edddd9df8eb9bc45e9d6f27decd"
server_key_iv: "3993a5bf504ef6eabc3724bf399d889c"
server_key_version: "encryptionkey@0"
I20260927 14:59:49.767059 16104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "52a91d287a1e40c58f1f7f7951c20648"
format_stamp: "Formatted at 2026-09-27 14:59:49 on dist-test-slave-pz0h"
server_key: "50583edddd9df8eb9bc45e9d6f27decd"
server_key_iv: "3993a5bf504ef6eabc3724bf399d889c"
server_key_version: "encryptionkey@0"
I20260927 14:59:49.771029 16104 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 14:59:49.773754 16119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:49.775228 16104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 14:59:49.775373 16104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "52a91d287a1e40c58f1f7f7951c20648"
format_stamp: "Formatted at 2026-09-27 14:59:49 on dist-test-slave-pz0h"
server_key: "50583edddd9df8eb9bc45e9d6f27decd"
server_key_iv: "3993a5bf504ef6eabc3724bf399d889c"
server_key_version: "encryptionkey@0"
I20260927 14:59:49.775497 16104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 14:59:49.801867 16104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 14:59:49.805171 16104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 14:59:49.805361 16104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 14:59:49.805959 16104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 14:59:49.806844 16104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 14:59:49.806893 16104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:49.806948 16104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 14:59:49.806975 16104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 14:59:49.815575 16104 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41517
I20260927 14:59:49.815658 16232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41517 every 8 connection(s)
I20260927 14:59:49.816530 16104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 14:59:49.826618 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 16104
I20260927 14:59:49.826728 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/wal/instance
I20260927 14:59:49.827077 392 external_mini_cluster.cc:1428] Setting key 7a7214f7f7b7d2c1b1ee74b7450df4e7
Sep 27 14:59:49 dist-test-slave-pz0h krb5kdc[15729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 14:59:49.831802 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:49.818516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53493 (local address 127.0.98.62:34767)
0927 14:59:49.818688 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:49.818693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:49.819690 (+ 997us) server_negotiation.cc:407] Connection header received
0927 14:59:49.820952 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:49.820955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:49.821008 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:49.821088 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:49.822407 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:49.823077 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:49.824313 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:49.824556 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:49.824660 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:49.828253 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:49.828257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:49.828260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:49.828291 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:49.829676 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:49.830203 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:49.830207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:49.830209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:49.830287 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:49.830624 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:49.830628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:49.830629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:49.830865 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:49.830956 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:49.831501 (+ 545us) server_negotiation.cc:299] Negotiation successful
0927 14:59:49.831641 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 14:59:49.832660 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:49.818976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:53493)
0927 14:59:49.819503 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:49.819548 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:49.820736 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:49.821305 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:49.821313 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:49.821693 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 14:59:49.822231 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:49.822247 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:49.823198 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:49.823201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:49.824151 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:49.824163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:49.824846 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:49.824852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:49.825119 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:49.826275 (+ 1156us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 14:59:49.826299 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 14:59:49.828065 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 14:59:49.829800 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:49.829807 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:49.829825 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:49.830106 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:49.830371 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 14:59:49.830374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 14:59:49.830376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 14:59:49.830527 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 14:59:49.830966 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 14:59:49.830974 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 14:59:49.831260 (+ 286us) client_negotiation.cc:769] Sending connection context
0927 14:59:49.831440 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 14:59:49.831815 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":152,"threads_started":1}
I20260927 14:59:49.833743 16233 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34767
I20260927 14:59:49.833958 16233 heartbeater.cc:464] Registering TS with master...
I20260927 14:59:49.834417 16233 heartbeater.cc:511] Master 127.0.98.62:34767 requested a full tablet report, sending...
I20260927 14:59:49.835462 15774 ts_manager.cc:194] Registered new tserver with Master: 52a91d287a1e40c58f1f7f7951c20648 (127.0.98.3:41517)
I20260927 14:59:49.836000 15774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53493
I20260927 14:59:49.841223 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 14:59:49.850591 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:49.843308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57616 (local address 127.0.98.62:34767)
0927 14:59:49.843477 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:49.843482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:49.843599 (+ 117us) server_negotiation.cc:407] Connection header received
0927 14:59:49.843735 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:49.843739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:49.843782 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:49.843849 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 14:59:49.844543 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:49.845063 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:49.845890 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:49.846103 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:49.846198 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:49.847553 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 14:59:49.847557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 14:59:49.847559 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 14:59:49.847584 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 14:59:49.848943 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:49.849329 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:49.849333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:49.849335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:49.849420 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 14:59:49.849663 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 14:59:49.849666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 14:59:49.849668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 14:59:49.849945 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 14:59:49.850067 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:49.850313 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 14:59:49.850433 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 14:59:49.853614 15774 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 14:59:49.857800 15774 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 14:59:49.878959 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:49.873918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41930 (local address 127.0.98.3:41517)
0927 14:59:49.874229 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:49.874240 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:49.874445 (+ 205us) server_negotiation.cc:407] Connection header received
0927 14:59:49.874612 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:49.874618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:49.874844 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:49.874989 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:59:49.875220 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:49.876096 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:49.877523 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:49.878191 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:49.878300 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:49.878460 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:49.878650 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 14:59:49.878762 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":122,"threads_started":1}
I20260927 14:59:49.878957 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:49.873490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41517 (local address 127.0.0.1:41930)
0927 14:59:49.874358 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:49.874371 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:49.874478 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:49.874988 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:49.874991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:49.875009 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:59:49.875112 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:49.875117 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:49.876219 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:49.876223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:49.877405 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:49.877415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:49.878316 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:49.878320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:49.878437 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:49.878485 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 14:59:49.878616 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 14:59:49.878768 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"mutex_wait_us":36,"thread_start_us":87,"threads_started":1}
I20260927 14:59:49.879534 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:49.873154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39060 (local address 127.0.98.2:33955)
0927 14:59:49.873948 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:49.873960 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:49.873986 (+ 26us) server_negotiation.cc:407] Connection header received
0927 14:59:49.874070 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:49.874075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:49.874299 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:49.874450 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:59:49.874750 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:49.875700 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:49.877824 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:49.878492 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:49.878592 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:49.878729 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:49.879288 (+ 559us) server_negotiation.cc:299] Negotiation successful
0927 14:59:49.879393 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":102,"threads_started":1}
I20260927 14:59:49.879534 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:49.873576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46197 (local address 127.0.0.1:56960)
0927 14:59:49.874666 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:49.874678 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:49.874753 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:49.875396 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:49.875399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:49.875415 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:59:49.875520 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:49.875526 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:49.876766 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:49.876770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:49.877850 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:49.877858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:49.879140 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:49.879147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:49.879252 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:49.879277 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 14:59:49.879344 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 14:59:49.879413 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"thread_start_us":64,"threads_started":1}
I20260927 14:59:49.879197 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:49.872998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33955 (local address 127.0.0.1:39060)
0927 14:59:49.873420 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:49.873476 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:49.873605 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:49.874457 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:49.874463 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:49.874486 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:59:49.874635 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:49.874642 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:49.876353 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:49.876356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:49.877684 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:49.877695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:49.878818 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:49.878822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:49.878908 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:49.878928 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 14:59:49.878992 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 14:59:49.879058 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":32,"thread_start_us":107,"threads_started":1}
I20260927 14:59:49.879926 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:49.873924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56960 (local address 127.0.98.1:46197)
0927 14:59:49.874391 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:49.874398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:49.874850 (+ 452us) server_negotiation.cc:407] Connection header received
0927 14:59:49.874930 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:49.874935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:49.875116 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:49.875231 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:59:49.875636 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:49.876628 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:49.877970 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:49.878728 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:49.878856 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:49.878986 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:49.879687 (+ 701us) server_negotiation.cc:299] Negotiation successful
0927 14:59:49.879797 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":120,"threads_started":1}
I20260927 14:59:49.882449 16033 tablet_service.cc:1467] Processing CreateTablet for tablet b3ca96acd4144bcfbb71e3cab6d55d91 (DEFAULT_TABLE table=test-table [id=96a7fd8c372b42f291ee69b7a8950056]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:59:49.882534 16167 tablet_service.cc:1467] Processing CreateTablet for tablet b3ca96acd4144bcfbb71e3cab6d55d91 (DEFAULT_TABLE table=test-table [id=96a7fd8c372b42f291ee69b7a8950056]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:59:49.882534 15899 tablet_service.cc:1467] Processing CreateTablet for tablet b3ca96acd4144bcfbb71e3cab6d55d91 (DEFAULT_TABLE table=test-table [id=96a7fd8c372b42f291ee69b7a8950056]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 14:59:49.883599 16033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3ca96acd4144bcfbb71e3cab6d55d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:59:49.883618 15899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3ca96acd4144bcfbb71e3cab6d55d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:59:49.883626 16167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3ca96acd4144bcfbb71e3cab6d55d91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 14:59:49.890578 16250 tablet_bootstrap.cc:492] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238: Bootstrap starting.
I20260927 14:59:49.891956 16251 tablet_bootstrap.cc:492] T b3ca96acd4144bcfbb71e3cab6d55d91 P 2e934c9fbd86496fa240f8cbfce7c05b: Bootstrap starting.
I20260927 14:59:49.893168 16250 tablet_bootstrap.cc:654] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238: Neither blocks nor log segments found. Creating new log.
I20260927 14:59:49.893864 16251 tablet_bootstrap.cc:654] T b3ca96acd4144bcfbb71e3cab6d55d91 P 2e934c9fbd86496fa240f8cbfce7c05b: Neither blocks nor log segments found. Creating new log.
I20260927 14:59:49.893939 16250 log.cc:824] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238: Log is configured to *not* fsync() on all Append() calls
I20260927 14:59:49.894606 16251 log.cc:824] T b3ca96acd4144bcfbb71e3cab6d55d91 P 2e934c9fbd86496fa240f8cbfce7c05b: Log is configured to *not* fsync() on all Append() calls
I20260927 14:59:49.894748 16252 tablet_bootstrap.cc:492] T b3ca96acd4144bcfbb71e3cab6d55d91 P 52a91d287a1e40c58f1f7f7951c20648: Bootstrap starting.
I20260927 14:59:49.896390 16250 tablet_bootstrap.cc:492] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238: No bootstrap required, opened a new log
I20260927 14:59:49.896389 16251 tablet_bootstrap.cc:492] T b3ca96acd4144bcfbb71e3cab6d55d91 P 2e934c9fbd86496fa240f8cbfce7c05b: No bootstrap required, opened a new log
I20260927 14:59:49.896600 16251 ts_tablet_manager.cc:1397] T b3ca96acd4144bcfbb71e3cab6d55d91 P 2e934c9fbd86496fa240f8cbfce7c05b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 14:59:49.896600 16250 ts_tablet_manager.cc:1397] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 14:59:49.896831 16252 tablet_bootstrap.cc:654] T b3ca96acd4144bcfbb71e3cab6d55d91 P 52a91d287a1e40c58f1f7f7951c20648: Neither blocks nor log segments found. Creating new log.
I20260927 14:59:49.897814 16252 log.cc:824] T b3ca96acd4144bcfbb71e3cab6d55d91 P 52a91d287a1e40c58f1f7f7951c20648: Log is configured to *not* fsync() on all Append() calls
I20260927 14:59:49.899622 16252 tablet_bootstrap.cc:492] T b3ca96acd4144bcfbb71e3cab6d55d91 P 52a91d287a1e40c58f1f7f7951c20648: No bootstrap required, opened a new log
I20260927 14:59:49.899813 16252 ts_tablet_manager.cc:1397] T b3ca96acd4144bcfbb71e3cab6d55d91 P 52a91d287a1e40c58f1f7f7951c20648: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 14:59:49.900689 16250 raft_consensus.cc:348] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b093d925204400bbc7da11cd8629238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33955 } } peers { permanent_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46197 } } peers { permanent_uuid: "52a91d287a1e40c58f1f7f7951c20648" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41517 } }
I20260927 14:59:49.900997 16250 raft_consensus.cc:374] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:59:49.901049 16250 raft_consensus.cc:729] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b093d925204400bbc7da11cd8629238, State: Initialized, Role: FOLLOWER
I20260927 14:59:49.900918 16251 raft_consensus.cc:348] T b3ca96acd4144bcfbb71e3cab6d55d91 P 2e934c9fbd86496fa240f8cbfce7c05b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b093d925204400bbc7da11cd8629238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33955 } } peers { permanent_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46197 } } peers { permanent_uuid: "52a91d287a1e40c58f1f7f7951c20648" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41517 } }
I20260927 14:59:49.901129 16251 raft_consensus.cc:374] T b3ca96acd4144bcfbb71e3cab6d55d91 P 2e934c9fbd86496fa240f8cbfce7c05b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:59:49.901196 16251 raft_consensus.cc:729] T b3ca96acd4144bcfbb71e3cab6d55d91 P 2e934c9fbd86496fa240f8cbfce7c05b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e934c9fbd86496fa240f8cbfce7c05b, State: Initialized, Role: FOLLOWER
I20260927 14:59:49.901583 16251 consensus_queue.cc:260] T b3ca96acd4144bcfbb71e3cab6d55d91 P 2e934c9fbd86496fa240f8cbfce7c05b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b093d925204400bbc7da11cd8629238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33955 } } peers { permanent_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46197 } } peers { permanent_uuid: "52a91d287a1e40c58f1f7f7951c20648" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41517 } }
I20260927 14:59:49.901583 16250 consensus_queue.cc:260] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b093d925204400bbc7da11cd8629238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33955 } } peers { permanent_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46197 } } peers { permanent_uuid: "52a91d287a1e40c58f1f7f7951c20648" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41517 } }
I20260927 14:59:49.902648 16250 ts_tablet_manager.cc:1428] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 14:59:49.902648 16251 ts_tablet_manager.cc:1428] T b3ca96acd4144bcfbb71e3cab6d55d91 P 2e934c9fbd86496fa240f8cbfce7c05b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 14:59:49.903048 16099 heartbeater.cc:503] Master 127.0.98.62:34767 was elected leader, sending a full tablet report...
I20260927 14:59:49.903048 15965 heartbeater.cc:503] Master 127.0.98.62:34767 was elected leader, sending a full tablet report...
I20260927 14:59:49.903225 16252 raft_consensus.cc:348] T b3ca96acd4144bcfbb71e3cab6d55d91 P 52a91d287a1e40c58f1f7f7951c20648 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b093d925204400bbc7da11cd8629238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33955 } } peers { permanent_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46197 } } peers { permanent_uuid: "52a91d287a1e40c58f1f7f7951c20648" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41517 } }
I20260927 14:59:49.903448 16252 raft_consensus.cc:374] T b3ca96acd4144bcfbb71e3cab6d55d91 P 52a91d287a1e40c58f1f7f7951c20648 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 14:59:49.903504 16252 raft_consensus.cc:729] T b3ca96acd4144bcfbb71e3cab6d55d91 P 52a91d287a1e40c58f1f7f7951c20648 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a91d287a1e40c58f1f7f7951c20648, State: Initialized, Role: FOLLOWER
I20260927 14:59:49.903929 16252 consensus_queue.cc:260] T b3ca96acd4144bcfbb71e3cab6d55d91 P 52a91d287a1e40c58f1f7f7951c20648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b093d925204400bbc7da11cd8629238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33955 } } peers { permanent_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46197 } } peers { permanent_uuid: "52a91d287a1e40c58f1f7f7951c20648" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41517 } }
I20260927 14:59:49.905861 16252 ts_tablet_manager.cc:1428] T b3ca96acd4144bcfbb71e3cab6d55d91 P 52a91d287a1e40c58f1f7f7951c20648: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 14:59:49.906356 16233 heartbeater.cc:503] Master 127.0.98.62:34767 was elected leader, sending a full tablet report...
W20260927 14:59:50.067881 16234 tablet.cc:2369] T b3ca96acd4144bcfbb71e3cab6d55d91 P 52a91d287a1e40c58f1f7f7951c20648: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 14:59:50.078832 16100 tablet.cc:2369] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 14:59:50.102255 15966 tablet.cc:2369] T b3ca96acd4144bcfbb71e3cab6d55d91 P 2e934c9fbd86496fa240f8cbfce7c05b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 14:59:50.153717 16257 raft_consensus.cc:482] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 14:59:50.153940 16257 raft_consensus.cc:504] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b093d925204400bbc7da11cd8629238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33955 } } peers { permanent_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46197 } } peers { permanent_uuid: "52a91d287a1e40c58f1f7f7951c20648" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41517 } }
I20260927 14:59:50.155177 16257 leader_election.cc:288] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e934c9fbd86496fa240f8cbfce7c05b (127.0.98.1:46197), 52a91d287a1e40c58f1f7f7951c20648 (127.0.98.3:41517)
I20260927 14:59:50.160710 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:50.155644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32847 (local address 127.0.98.3:41517)
0927 14:59:50.155813 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:50.155816 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:50.155990 (+ 174us) server_negotiation.cc:407] Connection header received
0927 14:59:50.156095 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:50.156099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:50.156155 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:50.156243 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:59:50.156643 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:50.157438 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:50.159295 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:50.159944 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:50.160081 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:50.160153 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:50.160471 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 14:59:50.160563 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 14:59:50.160710 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:50.155613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34417 (local address 127.0.98.1:46197)
0927 14:59:50.155747 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:50.155752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:50.155970 (+ 218us) server_negotiation.cc:407] Connection header received
0927 14:59:50.156096 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:50.156099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:50.156155 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:50.156243 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 14:59:50.156643 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:50.157439 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:50.159295 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:50.159959 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:50.160081 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:50.160153 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:50.160471 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 14:59:50.160563 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 14:59:50.160799 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:50.155470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41517 (local address 127.0.98.2:32847)
0927 14:59:50.155826 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:50.155841 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:50.155996 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:50.156292 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:50.156295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:50.156324 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:59:50.156516 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:50.156521 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:50.157579 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:50.157583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:50.159120 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:50.159137 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:50.160134 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:50.160138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:50.160274 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:50.160321 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 14:59:50.160421 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 14:59:50.160557 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":202,"spinlock_wait_cycles":17792,"thread_start_us":92,"threads_started":1}
I20260927 14:59:50.160801 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:50.155473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46197 (local address 127.0.98.2:34417)
0927 14:59:50.155846 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:50.155858 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 14:59:50.155996 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 14:59:50.156292 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 14:59:50.156295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 14:59:50.156324 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 14:59:50.156514 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:50.156521 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:50.157579 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:50.157583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:50.159120 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 14:59:50.159137 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 14:59:50.160134 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 14:59:50.160138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 14:59:50.160278 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:50.160323 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 14:59:50.160422 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 14:59:50.160557 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":232,"spinlock_wait_cycles":14592,"thread_start_us":65,"threads_started":1}
I20260927 14:59:50.161556 16187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ca96acd4144bcfbb71e3cab6d55d91" candidate_uuid: "4b093d925204400bbc7da11cd8629238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a91d287a1e40c58f1f7f7951c20648" is_pre_election: true
I20260927 14:59:50.161911 16187 raft_consensus.cc:2454] T b3ca96acd4144bcfbb71e3cab6d55d91 P 52a91d287a1e40c58f1f7f7951c20648 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b093d925204400bbc7da11cd8629238 in term 0.
I20260927 14:59:50.161876 15919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ca96acd4144bcfbb71e3cab6d55d91" candidate_uuid: "4b093d925204400bbc7da11cd8629238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" is_pre_election: true
I20260927 14:59:50.162274 15919 raft_consensus.cc:2454] T b3ca96acd4144bcfbb71e3cab6d55d91 P 2e934c9fbd86496fa240f8cbfce7c05b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b093d925204400bbc7da11cd8629238 in term 0.
I20260927 14:59:50.162456 15988 leader_election.cc:302] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b093d925204400bbc7da11cd8629238, 52a91d287a1e40c58f1f7f7951c20648; no voters:
I20260927 14:59:50.162751 16257 raft_consensus.cc:2781] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 14:59:50.162878 16257 raft_consensus.cc:482] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 14:59:50.162945 16257 raft_consensus.cc:3037] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:59:50.164192 16257 raft_consensus.cc:504] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b093d925204400bbc7da11cd8629238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33955 } } peers { permanent_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46197 } } peers { permanent_uuid: "52a91d287a1e40c58f1f7f7951c20648" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41517 } }
I20260927 14:59:50.164731 16257 leader_election.cc:288] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238 [CANDIDATE]: Term 1 election: Requested vote from peers 2e934c9fbd86496fa240f8cbfce7c05b (127.0.98.1:46197), 52a91d287a1e40c58f1f7f7951c20648 (127.0.98.3:41517)
I20260927 14:59:50.165105 15919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ca96acd4144bcfbb71e3cab6d55d91" candidate_uuid: "4b093d925204400bbc7da11cd8629238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e934c9fbd86496fa240f8cbfce7c05b"
I20260927 14:59:50.165226 16187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ca96acd4144bcfbb71e3cab6d55d91" candidate_uuid: "4b093d925204400bbc7da11cd8629238" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a91d287a1e40c58f1f7f7951c20648"
I20260927 14:59:50.165303 15919 raft_consensus.cc:3037] T b3ca96acd4144bcfbb71e3cab6d55d91 P 2e934c9fbd86496fa240f8cbfce7c05b [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:59:50.165370 16187 raft_consensus.cc:3037] T b3ca96acd4144bcfbb71e3cab6d55d91 P 52a91d287a1e40c58f1f7f7951c20648 [term 0 FOLLOWER]: Advancing to term 1
I20260927 14:59:50.166443 15919 raft_consensus.cc:2454] T b3ca96acd4144bcfbb71e3cab6d55d91 P 2e934c9fbd86496fa240f8cbfce7c05b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b093d925204400bbc7da11cd8629238 in term 1.
I20260927 14:59:50.166440 16187 raft_consensus.cc:2454] T b3ca96acd4144bcfbb71e3cab6d55d91 P 52a91d287a1e40c58f1f7f7951c20648 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b093d925204400bbc7da11cd8629238 in term 1.
I20260927 14:59:50.166746 15988 leader_election.cc:302] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b093d925204400bbc7da11cd8629238, 52a91d287a1e40c58f1f7f7951c20648; no voters:
I20260927 14:59:50.166985 16257 raft_consensus.cc:2781] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 14:59:50.167220 16257 raft_consensus.cc:686] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238 [term 1 LEADER]: Becoming Leader. State: Replica: 4b093d925204400bbc7da11cd8629238, State: Running, Role: LEADER
I20260927 14:59:50.167605 16257 consensus_queue.cc:237] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b093d925204400bbc7da11cd8629238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33955 } } peers { permanent_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46197 } } peers { permanent_uuid: "52a91d287a1e40c58f1f7f7951c20648" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41517 } }
I20260927 14:59:50.170923 15772 catalog_manager.cc:5495] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b093d925204400bbc7da11cd8629238 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4b093d925204400bbc7da11cd8629238" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b093d925204400bbc7da11cd8629238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52a91d287a1e40c58f1f7f7951c20648" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41517 } health_report { overall_health: UNKNOWN } } }
I20260927 14:59:50.246266 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:50.242020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39062 (local address 127.0.98.2:33955)
0927 14:59:50.242148 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 14:59:50.242151 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 14:59:50.242166 (+ 15us) server_negotiation.cc:407] Connection header received
0927 14:59:50.242246 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 14:59:50.242248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 14:59:50.242307 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 14:59:50.242398 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 14:59:50.242682 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:50.243372 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:50.244362 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 14:59:50.244599 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 14:59:50.244698 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 14:59:50.245093 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 14:59:50.245585 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 14:59:50.245692 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 14:59:50.245976 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 14:59:50.246094 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 14:59:50.253505 16187 raft_consensus.cc:1261] T b3ca96acd4144bcfbb71e3cab6d55d91 P 52a91d287a1e40c58f1f7f7951c20648 [term 1 FOLLOWER]: Refusing update from remote peer 4b093d925204400bbc7da11cd8629238: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:59:50.253502 15919 raft_consensus.cc:1261] T b3ca96acd4144bcfbb71e3cab6d55d91 P 2e934c9fbd86496fa240f8cbfce7c05b [term 1 FOLLOWER]: Refusing update from remote peer 4b093d925204400bbc7da11cd8629238: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 14:59:50.254477 16257 consensus_queue.cc:1035] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:59:50.254981 16261 consensus_queue.cc:1035] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52a91d287a1e40c58f1f7f7951c20648" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 14:59:50.264894 16265 mvcc.cc:205] Tried to move back new op lower bound from 7333974795270017024 to 7333974794929332224. Current Snapshot: MvccSnapshot[applied={T|T < 7333974795270017024}]
I20260927 14:59:50.273550 15772 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:59:50.273782 15772 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 14:59:50.276859 15772 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a: Sending DeleteTablet for 3 replicas of tablet b3ca96acd4144bcfbb71e3cab6d55d91
I20260927 14:59:50.277848 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 15836
I20260927 14:59:50.277875 16167 tablet_service.cc:1514] Processing DeleteTablet for tablet b3ca96acd4144bcfbb71e3cab6d55d91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:59:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41930
I20260927 14:59:50.277877 15899 tablet_service.cc:1514] Processing DeleteTablet for tablet b3ca96acd4144bcfbb71e3cab6d55d91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:59:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56960
I20260927 14:59:50.278600 16033 tablet_service.cc:1514] Processing DeleteTablet for tablet b3ca96acd4144bcfbb71e3cab6d55d91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:59:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39060
I20260927 14:59:50.279404 16274 tablet_replica.cc:330] T b3ca96acd4144bcfbb71e3cab6d55d91 P 52a91d287a1e40c58f1f7f7951c20648: stopping tablet replica
I20260927 14:59:50.279722 16274 raft_consensus.cc:2229] T b3ca96acd4144bcfbb71e3cab6d55d91 P 52a91d287a1e40c58f1f7f7951c20648 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 14:59:50.280078 16274 raft_consensus.cc:2258] T b3ca96acd4144bcfbb71e3cab6d55d91 P 52a91d287a1e40c58f1f7f7951c20648 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:59:50.280511 16275 tablet_replica.cc:330] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238: stopping tablet replica
I20260927 14:59:50.280807 16275 raft_consensus.cc:2229] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238 [term 1 LEADER]: Raft consensus shutting down.
I20260927 14:59:50.281252 16275 raft_consensus.cc:2258] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 14:59:50.282035 16274 ts_tablet_manager.cc:1905] T b3ca96acd4144bcfbb71e3cab6d55d91 P 52a91d287a1e40c58f1f7f7951c20648: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:59:50.282727 16275 ts_tablet_manager.cc:1905] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 14:59:50.285586 16274 ts_tablet_manager.cc:1918] T b3ca96acd4144bcfbb71e3cab6d55d91 P 52a91d287a1e40c58f1f7f7951c20648: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:59:50.285714 16275 ts_tablet_manager.cc:1918] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 14:59:50.285807 16275 log.cc:1196] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/wal/wals/b3ca96acd4144bcfbb71e3cab6d55d91
I20260927 14:59:50.285902 16274 log.cc:1196] T b3ca96acd4144bcfbb71e3cab6d55d91 P 52a91d287a1e40c58f1f7f7951c20648: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/wal/wals/b3ca96acd4144bcfbb71e3cab6d55d91
I20260927 14:59:50.286142 16275 ts_tablet_manager.cc:1939] T b3ca96acd4144bcfbb71e3cab6d55d91 P 4b093d925204400bbc7da11cd8629238: Deleting consensus metadata
W20260927 14:59:50.286288 15758 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46197 (error 108)
I20260927 14:59:50.286708 16274 ts_tablet_manager.cc:1939] T b3ca96acd4144bcfbb71e3cab6d55d91 P 52a91d287a1e40c58f1f7f7951c20648: Deleting consensus metadata
I20260927 14:59:50.287554 15759 catalog_manager.cc:4841] TS 4b093d925204400bbc7da11cd8629238 (127.0.98.2:33955): tablet b3ca96acd4144bcfbb71e3cab6d55d91 (table test-table [id=96a7fd8c372b42f291ee69b7a8950056]) successfully deleted
I20260927 14:59:50.287674 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 15970
I20260927 14:59:50.287763 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 14:59:50.287424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46197 (local address 127.0.0.1:56972)
0927 14:59:50.287613 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 14:59:50.287682 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260927 14:59:50.288825 15760 catalog_manager.cc:4841] TS 52a91d287a1e40c58f1f7f7951c20648 (127.0.98.3:41517): tablet b3ca96acd4144bcfbb71e3cab6d55d91 (table test-table [id=96a7fd8c372b42f291ee69b7a8950056]) successfully deleted
W20260927 14:59:50.289144 15758 catalog_manager.cc:4584] TS 2e934c9fbd86496fa240f8cbfce7c05b (127.0.98.1:46197): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b3ca96acd4144bcfbb71e3cab6d55d91: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46197: connect: Connection refused (error 111)
I20260927 14:59:50.293227 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 16104
I20260927 14:59:50.298332 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 15742
I20260927 15:00:10.304409 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34767
--webserver_interface=127.0.98.62
--webserver_port=39615
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:00:10.429677 16277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:10.430016 16277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:10.430059 16277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:10.434286 16277 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 15:00:10.434355 16277 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 15:00:10.434371 16277 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 15:00:10.434384 16277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:00:10.434402 16277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:10.434420 16277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:10.434434 16277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:00:10.434448 16277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:00:10.440079 16277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:10.441234 16277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:10.442345 16277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:00:10.455834 16284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:10.455884 16277 server_base.cc:1034] running on GCE node
W20260927 15:00:10.455840 16286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:10.455883 16283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:10.457082 16277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:10.458431 16277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:10.459645 16277 hybrid_clock.cc:648] HybridClock initialized: now 1790521210459591 us; error 70 us; skew 500 ppm
Sep 27 15:00:10 dist-test-slave-pz0h krb5kdc[15729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:00:10.462836 16277 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:00:10.463904 16277 webserver.cc:460] Webserver started at http://127.0.98.62:39615/ using document root <none> and password file <none>
I20260927 15:00:10.464534 16277 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:10.464617 16277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:10.469043 16277 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:00:10.471544 16293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:10.473342 16277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:00:10.473485 16277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "2b08eee532aa45d88003af67633d057a"
format_stamp: "Formatted at 2026-09-27 14:59:48 on dist-test-slave-pz0h"
server_key: "99e5367e5fedfb1a4eeabaab4fb95e20"
server_key_iv: "7e3be957b1ce1e1cbf40e419ba2cd936"
server_key_version: "encryptionkey@0"
I20260927 15:00:10.474143 16277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:10.487825 16277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:10.491217 16277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:10.491427 16277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:10.498723 16277 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34767
I20260927 15:00:10.499053 16345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34767 every 8 connection(s)
I20260927 15:00:10.500021 16277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:00:10.501681 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 16277
I20260927 15:00:10.502084 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46197
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34767
--webserver_port=37049
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:00:10.506328 16346 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:00:10.509150 16346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a: Bootstrap starting.
I20260927 15:00:10.532482 16346 log.cc:824] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a: Log is configured to *not* fsync() on all Append() calls
I20260927 15:00:10.544848 16346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:00:10.545351 16346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a: Bootstrap complete.
I20260927 15:00:10.551086 16346 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b08eee532aa45d88003af67633d057a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } }
I20260927 15:00:10.551702 16346 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b08eee532aa45d88003af67633d057a, State: Initialized, Role: FOLLOWER
I20260927 15:00:10.552338 16346 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b08eee532aa45d88003af67633d057a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } }
I20260927 15:00:10.552500 16346 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:00:10.552577 16346 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:00:10.552680 16346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:00:10.555495 16346 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b08eee532aa45d88003af67633d057a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } }
I20260927 15:00:10.555895 16346 leader_election.cc:302] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b08eee532aa45d88003af67633d057a; no voters:
I20260927 15:00:10.556236 16346 leader_election.cc:288] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 15:00:10.556403 16350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [term 2 FOLLOWER]: Leader election won for term 2
I20260927 15:00:10.556663 16350 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [term 2 LEADER]: Becoming Leader. State: Replica: 2b08eee532aa45d88003af67633d057a, State: Running, Role: LEADER
I20260927 15:00:10.557004 16350 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b08eee532aa45d88003af67633d057a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } }
I20260927 15:00:10.557552 16346 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:00:10.558127 16350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b08eee532aa45d88003af67633d057a. Latest consensus state: current_term: 2 leader_uuid: "2b08eee532aa45d88003af67633d057a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b08eee532aa45d88003af67633d057a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } } }
I20260927 15:00:10.558259 16350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [sys.catalog]: This master's current role is: LEADER
I20260927 15:00:10.558625 16356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:00:10.558728 16351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2b08eee532aa45d88003af67633d057a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b08eee532aa45d88003af67633d057a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } } }
I20260927 15:00:10.558933 16351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [sys.catalog]: This master's current role is: LEADER
I20260927 15:00:10.564205 16356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:00:10.564962 16356 catalog_manager.cc:1259] Loaded cluster ID: e0a207cee683435db0e36621f7e4cac1
I20260927 15:00:10.565044 16356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:00:10.567603 16356 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:00:10.568452 16356 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a: Loaded TSK: 0
I20260927 15:00:10.590155 16356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a: Generated new TSK 1
I20260927 15:00:10.590481 16356 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:00:10.646291 16348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:10.646596 16348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:10.646646 16348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:10.651085 16348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:10.651160 16348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:10.651229 16348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:00:10.656910 16348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46197
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37049
--tserver_master_addrs=127.0.98.62:34767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:10.658110 16348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:10.659340 16348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:00:10.669821 16376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:10.669831 16374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:10.669812 16373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:10.670248 16348 server_base.cc:1034] running on GCE node
I20260927 15:00:10.670650 16348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:10.671397 16348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:10.672595 16348 hybrid_clock.cc:648] HybridClock initialized: now 1790521210672557 us; error 57 us; skew 500 ppm
Sep 27 15:00:10 dist-test-slave-pz0h krb5kdc[15729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:00:10.675587 16348 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 15:00:10.676793 16348 webserver.cc:460] Webserver started at http://127.0.98.1:37049/ using document root <none> and password file <none>
I20260927 15:00:10.677438 16348 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:10.677537 16348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:10.682224 16348 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:00:10.684805 16383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:10.686300 16348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 15:00:10.686450 16348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "2e934c9fbd86496fa240f8cbfce7c05b"
format_stamp: "Formatted at 2026-09-27 14:59:49 on dist-test-slave-pz0h"
server_key: "4473e166550f139ffcc405c8b25b5b8c"
server_key_iv: "b4412227f296a39a1110fe733686bf39"
server_key_version: "encryptionkey@0"
I20260927 15:00:10.687103 16348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:10.699033 16348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:10.702378 16348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:10.702593 16348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:10.703275 16348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:00:10.704494 16390 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 15:00:10.708818 16348 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 15:00:10.708922 16348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260927 15:00:10.709002 16348 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 15:00:10.710983 16348 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 15:00:10.711040 16348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260927 15:00:10.711373 16390 tablet_bootstrap.cc:492] T b3ca96acd4144bcfbb71e3cab6d55d91 P 2e934c9fbd86496fa240f8cbfce7c05b: Bootstrap starting.
I20260927 15:00:10.720732 16348 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46197
I20260927 15:00:10.720841 16497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46197 every 8 connection(s)
I20260927 15:00:10.721760 16348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:00:10.730352 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 16348
I20260927 15:00:10.730826 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33955
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34767
--webserver_port=33521
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:00:10.731353 16390 log.cc:824] T b3ca96acd4144bcfbb71e3cab6d55d91 P 2e934c9fbd86496fa240f8cbfce7c05b: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:00:10 dist-test-slave-pz0h krb5kdc[15729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:00:10.739630 16390 tablet_bootstrap.cc:492] T b3ca96acd4144bcfbb71e3cab6d55d91 P 2e934c9fbd86496fa240f8cbfce7c05b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:00:10.740083 16390 tablet_bootstrap.cc:492] T b3ca96acd4144bcfbb71e3cab6d55d91 P 2e934c9fbd86496fa240f8cbfce7c05b: Bootstrap complete.
I20260927 15:00:10.741364 16390 ts_tablet_manager.cc:1397] T b3ca96acd4144bcfbb71e3cab6d55d91 P 2e934c9fbd86496fa240f8cbfce7c05b: Time spent bootstrapping tablet: real 0.030s user 0.019s sys 0.008s
I20260927 15:00:10.741464 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:10.724037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:50835)
0927 15:00:10.724710 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:10.724781 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:10.725941 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:10.727180 (+ 1239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:10.727190 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:10.727751 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:00:10.728457 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:10.728480 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:10.729851 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:10.729855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:10.731174 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:10.731186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:10.732253 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:10.732258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:10.732471 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:10.733167 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:00:10.733188 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:00:10.735121 (+ 1933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:00:10.737819 (+ 2698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:00:10.737825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:00:10.737837 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:00:10.738189 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:00:10.738626 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:00:10.738629 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:00:10.738631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:00:10.738834 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:00:10.739600 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:00:10.739607 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:00:10.739924 (+ 317us) client_negotiation.cc:769] Sending connection context
0927 15:00:10.740126 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 15:00:10.740479 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":119,"threads_started":1}
I20260927 15:00:10.742000 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:10.724064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50835 (local address 127.0.98.62:34767)
0927 15:00:10.724488 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:10.724508 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:10.724933 (+ 425us) server_negotiation.cc:407] Connection header received
0927 15:00:10.726384 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:10.726395 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:10.726722 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:10.727047 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:10.728684 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:10.729698 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:10.731337 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:10.732065 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:10.732876 (+ 811us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:10.735388 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:10.735396 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:10.735416 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:00:10.735457 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:10.737660 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:10.738323 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:10.738330 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:10.738337 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:10.738517 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:10.738944 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:10.738948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:10.738950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:10.739459 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:10.739620 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:10.740349 (+ 729us) server_negotiation.cc:299] Negotiation successful
0927 15:00:10.740700 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":143,"threads_started":1}
I20260927 15:00:10.744521 16498 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34767
I20260927 15:00:10.744110 16390 raft_consensus.cc:348] T b3ca96acd4144bcfbb71e3cab6d55d91 P 2e934c9fbd86496fa240f8cbfce7c05b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b093d925204400bbc7da11cd8629238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33955 } } peers { permanent_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46197 } } peers { permanent_uuid: "52a91d287a1e40c58f1f7f7951c20648" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41517 } }
I20260927 15:00:10.744702 16390 raft_consensus.cc:729] T b3ca96acd4144bcfbb71e3cab6d55d91 P 2e934c9fbd86496fa240f8cbfce7c05b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e934c9fbd86496fa240f8cbfce7c05b, State: Initialized, Role: FOLLOWER
I20260927 15:00:10.744803 16498 heartbeater.cc:464] Registering TS with master...
I20260927 15:00:10.745185 16390 consensus_queue.cc:260] T b3ca96acd4144bcfbb71e3cab6d55d91 P 2e934c9fbd86496fa240f8cbfce7c05b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b093d925204400bbc7da11cd8629238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33955 } } peers { permanent_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46197 } } peers { permanent_uuid: "52a91d287a1e40c58f1f7f7951c20648" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41517 } }
I20260927 15:00:10.745373 16498 heartbeater.cc:511] Master 127.0.98.62:34767 requested a full tablet report, sending...
I20260927 15:00:10.746133 16390 ts_tablet_manager.cc:1428] T b3ca96acd4144bcfbb71e3cab6d55d91 P 2e934c9fbd86496fa240f8cbfce7c05b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:00:10.748075 16310 ts_manager.cc:194] Registered new tserver with Master: 2e934c9fbd86496fa240f8cbfce7c05b (127.0.98.1:46197)
I20260927 15:00:10.751603 16310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50835
I20260927 15:00:10.754431 16498 heartbeater.cc:503] Master 127.0.98.62:34767 was elected leader, sending a full tablet report...
Sep 27 15:00:10 dist-test-slave-pz0h krb5kdc[15729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 15:00:10.762537 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:10.751104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46197 (local address 127.0.0.1:33656)
0927 15:00:10.751920 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:10.751965 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:10.752114 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:10.752634 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:10.752640 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:10.753008 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:00:10.753116 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:10.753123 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:10.754090 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:10.754094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:10.754765 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:10.754776 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:10.755490 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:10.755509 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:10.755630 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:10.756259 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:00:10.756274 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:00:10.757970 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:00:10.760451 (+ 2481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:00:10.760464 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:00:10.760469 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:00:10.760753 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:00:10.761099 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:00:10.761103 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:00:10.761104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:00:10.761231 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:00:10.761851 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:00:10.761857 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:00:10.762003 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 15:00:10.762146 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 15:00:10.762362 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":113,"threads_started":1}
I20260927 15:00:10.762913 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:10.751194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33656 (local address 127.0.98.1:46197)
0927 15:00:10.751616 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:10.751628 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:10.752081 (+ 453us) server_negotiation.cc:407] Connection header received
0927 15:00:10.752259 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:10.752264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:10.752477 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:10.752625 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:10.753270 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:10.753973 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:10.754893 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:10.755379 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:10.755489 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:10.758228 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:10.758251 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:10.758266 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:00:10.758299 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:10.760311 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:10.760887 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:10.760894 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:10.760896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:10.761006 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:10.761323 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:10.761326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:10.761328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:10.761700 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:10.761841 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:10.762251 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 15:00:10.762450 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":113,"threads_started":1}
I20260927 15:00:10.763985 16432 tablet_service.cc:1514] Processing DeleteTablet for tablet b3ca96acd4144bcfbb71e3cab6d55d91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 14:59:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33656
I20260927 15:00:10.764873 16507 tablet_replica.cc:330] stopping tablet replica
I20260927 15:00:10.765182 16507 raft_consensus.cc:2229] T b3ca96acd4144bcfbb71e3cab6d55d91 P 2e934c9fbd86496fa240f8cbfce7c05b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:00:10.765430 16507 raft_consensus.cc:2258] T b3ca96acd4144bcfbb71e3cab6d55d91 P 2e934c9fbd86496fa240f8cbfce7c05b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:00:10.766400 16507 ts_tablet_manager.cc:1905] T b3ca96acd4144bcfbb71e3cab6d55d91 P 2e934c9fbd86496fa240f8cbfce7c05b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:00:10.770119 16507 ts_tablet_manager.cc:1918] T b3ca96acd4144bcfbb71e3cab6d55d91 P 2e934c9fbd86496fa240f8cbfce7c05b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:00:10.770211 16507 log.cc:1196] T b3ca96acd4144bcfbb71e3cab6d55d91 P 2e934c9fbd86496fa240f8cbfce7c05b: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/wal/wals/b3ca96acd4144bcfbb71e3cab6d55d91
I20260927 15:00:10.770442 16507 ts_tablet_manager.cc:1939] T b3ca96acd4144bcfbb71e3cab6d55d91 P 2e934c9fbd86496fa240f8cbfce7c05b: Deleting consensus metadata
I20260927 15:00:10.772264 16294 catalog_manager.cc:4841] TS 2e934c9fbd86496fa240f8cbfce7c05b (127.0.98.1:46197): tablet b3ca96acd4144bcfbb71e3cab6d55d91 (table test-table [id=96a7fd8c372b42f291ee69b7a8950056]) successfully deleted
W20260927 15:00:10.859596 16503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:10.859889 16503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:10.859958 16503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:10.864337 16503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:10.864435 16503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:10.864521 16503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:00:10.870193 16503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33521
--tserver_master_addrs=127.0.98.62:34767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:10.871418 16503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:10.872478 16503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:00:10.882442 16512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:10.882474 16513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:10.882474 16515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:10.882654 16503 server_base.cc:1034] running on GCE node
I20260927 15:00:10.883271 16503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:10.883942 16503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:10.885104 16503 hybrid_clock.cc:648] HybridClock initialized: now 1790521210885089 us; error 35 us; skew 500 ppm
Sep 27 15:00:10 dist-test-slave-pz0h krb5kdc[15729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:00:10.892271 16503 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 15:00:10.893318 16503 webserver.cc:460] Webserver started at http://127.0.98.2:33521/ using document root <none> and password file <none>
I20260927 15:00:10.893888 16503 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:10.893972 16503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:10.898177 16503 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:00:10.900718 16522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:10.902230 16503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:00:10.902366 16503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "4b093d925204400bbc7da11cd8629238"
format_stamp: "Formatted at 2026-09-27 14:59:49 on dist-test-slave-pz0h"
server_key: "256fd6339acfeba332ca4f967668b99f"
server_key_iv: "8e0ba0bdc18e5bc67593adf95b6f54b4"
server_key_version: "encryptionkey@0"
I20260927 15:00:10.902999 16503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:10.917518 16503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:10.921098 16503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:10.921281 16503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:10.921854 16503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:00:10.922708 16503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:00:10.922757 16503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:10.922834 16503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:00:10.922868 16503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:10.931838 16503 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33955
I20260927 15:00:10.931862 16635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33955 every 8 connection(s)
I20260927 15:00:10.932893 16503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:00:10.939852 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 16503
I20260927 15:00:10.940366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41517
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34767
--webserver_port=44667
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:00:10 dist-test-slave-pz0h krb5kdc[15729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:00:10.951004 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:10.934741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48613 (local address 127.0.98.62:34767)
0927 15:00:10.934914 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:10.934918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:10.935770 (+ 852us) server_negotiation.cc:407] Connection header received
0927 15:00:10.937162 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:10.937167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:10.937232 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:10.937306 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:10.938657 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:10.939251 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:10.940930 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:10.941212 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:10.941318 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:10.944608 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:10.944613 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:10.944615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:00:10.944662 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:10.946493 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:10.947765 (+ 1272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:10.947770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:10.947771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:10.947908 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:10.948820 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:10.948826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:10.948827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:10.949155 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:10.949276 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:10.950635 (+ 1359us) server_negotiation.cc:299] Negotiation successful
0927 15:00:10.950842 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 15:00:10.952289 16638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:10.935154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:48613)
0927 15:00:10.935613 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:10.935657 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:10.936907 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:10.937513 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:10.937523 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:10.937926 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:00:10.938474 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:10.938490 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:10.939389 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:10.939392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:10.940766 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:10.940781 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:10.941517 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:10.941521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:10.941745 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:10.942427 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:00:10.942459 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:00:10.944323 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:00:10.946887 (+ 2564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:00:10.946914 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:00:10.946951 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:00:10.947571 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:00:10.948060 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:00:10.948064 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:00:10.948066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:00:10.948529 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:00:10.949357 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:00:10.949368 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:00:10.950145 (+ 777us) client_negotiation.cc:769] Sending connection context
0927 15:00:10.950486 (+ 341us) client_negotiation.cc:240] Negotiation successful
0927 15:00:10.951038 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":130,"threads_started":1}
I20260927 15:00:10.953804 16636 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34767
I20260927 15:00:10.954069 16636 heartbeater.cc:464] Registering TS with master...
I20260927 15:00:10.954627 16636 heartbeater.cc:511] Master 127.0.98.62:34767 requested a full tablet report, sending...
I20260927 15:00:10.955842 16310 ts_manager.cc:194] Registered new tserver with Master: 4b093d925204400bbc7da11cd8629238 (127.0.98.2:33955)
I20260927 15:00:10.956664 16310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48613
W20260927 15:00:11.060145 16639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:11.060427 16639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:11.060473 16639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:11.064761 16639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:11.064859 16639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:11.064954 16639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:00:11.070670 16639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44667
--tserver_master_addrs=127.0.98.62:34767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:11.071805 16639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:11.072819 16639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:00:11.082760 16645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:11.082777 16644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:11.082857 16639 server_base.cc:1034] running on GCE node
W20260927 15:00:11.082777 16647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:11.083453 16639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:11.084121 16639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:11.085310 16639 hybrid_clock.cc:648] HybridClock initialized: now 1790521211085250 us; error 100 us; skew 500 ppm
Sep 27 15:00:11 dist-test-slave-pz0h krb5kdc[15729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:00:11.088340 16639 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 15:00:11.089417 16639 webserver.cc:460] Webserver started at http://127.0.98.3:44667/ using document root <none> and password file <none>
I20260927 15:00:11.090063 16639 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:11.090149 16639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:11.094722 16639 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:00:11.097327 16654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:11.098703 16639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:00:11.098865 16639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "52a91d287a1e40c58f1f7f7951c20648"
format_stamp: "Formatted at 2026-09-27 14:59:49 on dist-test-slave-pz0h"
server_key: "50583edddd9df8eb9bc45e9d6f27decd"
server_key_iv: "3993a5bf504ef6eabc3724bf399d889c"
server_key_version: "encryptionkey@0"
I20260927 15:00:11.099478 16639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:11.115996 16639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:11.119520 16639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:11.119705 16639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:11.120292 16639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:00:11.121170 16639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:00:11.121218 16639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:11.121261 16639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:00:11.121282 16639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:11.130003 16639 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41517
I20260927 15:00:11.130069 16767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41517 every 8 connection(s)
I20260927 15:00:11.131080 16639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:00:11.138477 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 16639
Sep 27 15:00:11 dist-test-slave-pz0h krb5kdc[15729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:00:11.145794 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:11.132960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39767 (local address 127.0.98.62:34767)
0927 15:00:11.133098 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:11.133103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:11.133895 (+ 792us) server_negotiation.cc:407] Connection header received
0927 15:00:11.135284 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:11.135288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:11.135355 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:11.135433 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:11.136793 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:11.137327 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:11.138534 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:11.138836 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:11.138947 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:11.141800 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:11.141803 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:11.141805 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:00:11.141829 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:11.143443 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:11.144030 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:11.144034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:11.144036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:11.144115 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:11.144505 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:11.144509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:11.144511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:11.144706 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:11.144798 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:11.145495 (+ 697us) server_negotiation.cc:299] Negotiation successful
0927 15:00:11.145626 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:00:11.146865 16770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:11.133284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:39767)
0927 15:00:11.133733 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:11.133774 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:11.135029 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:11.135648 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:11.135656 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:11.136070 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:00:11.136618 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:11.136632 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:11.137463 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:11.137466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:11.138397 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:11.138408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:11.139007 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:11.139012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:11.139210 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:11.139931 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:00:11.139949 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:00:11.141625 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:00:11.143583 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:00:11.143589 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:00:11.143606 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:00:11.143899 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:00:11.144200 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:00:11.144203 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:00:11.144205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:00:11.144407 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:00:11.144810 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:00:11.144816 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:00:11.145184 (+ 368us) client_negotiation.cc:769] Sending connection context
0927 15:00:11.145420 (+ 236us) client_negotiation.cc:240] Negotiation successful
0927 15:00:11.145780 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":129,"threads_started":1}
I20260927 15:00:11.147184 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:11.139221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59002 (local address 127.0.98.62:34767)
0927 15:00:11.139496 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:11.139501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:11.139590 (+ 89us) server_negotiation.cc:407] Connection header received
0927 15:00:11.139720 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:11.139723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:11.139770 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:11.139859 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:11.140653 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:11.141245 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:11.142255 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:11.142551 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:11.142679 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:11.144039 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:11.144042 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:11.144044 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:00:11.144067 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:11.145536 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:11.146030 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:11.146033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:11.146035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:11.146110 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:11.146378 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:11.146380 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:11.146381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:11.146539 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:11.146699 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:11.146901 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 15:00:11.147033 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":73,"threads_started":1}
I20260927 15:00:11.148167 16768 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34767
I20260927 15:00:11.148418 16768 heartbeater.cc:464] Registering TS with master...
I20260927 15:00:11.148999 16768 heartbeater.cc:511] Master 127.0.98.62:34767 requested a full tablet report, sending...
I20260927 15:00:11.150035 16309 ts_manager.cc:194] Registered new tserver with Master: 52a91d287a1e40c58f1f7f7951c20648 (127.0.98.3:41517)
I20260927 15:00:11.150624 16309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39767
I20260927 15:00:11.160923 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:00:11.165212 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:11.161592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59006 (local address 127.0.98.62:34767)
0927 15:00:11.161716 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:11.161719 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:11.161883 (+ 164us) server_negotiation.cc:407] Connection header received
0927 15:00:11.162037 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:11.162039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:11.162155 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:11.162246 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:00:11.162438 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:11.163130 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:11.164069 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:11.164306 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:11.164403 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:11.164708 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:00:11.164860 (+ 152us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 15:00:11.165098 (+ 238us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59006: token expired
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 15:00:11.165292 16772 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59006: token expired
W20260927 15:00:11.165668 16236 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:34767)
I20260927 15:00:11.172797 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:11.166091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59016 (local address 127.0.98.62:34767)
0927 15:00:11.166193 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:11.166196 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:11.166208 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:00:11.166295 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:11.166297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:11.166335 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:11.166402 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:11.167047 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:11.167545 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:11.168320 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:11.168533 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:11.168607 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:11.169888 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:11.169890 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:11.169892 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:00:11.169916 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:11.171299 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:11.171764 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:11.171767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:11.171768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:11.171845 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:11.172056 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:11.172058 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:11.172059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:11.172244 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:11.172339 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:11.172508 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 15:00:11.172633 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 15:00:11.176301 16309 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:00:11.180065 16309 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:00:11.188786 16432 tablet_service.cc:1467] Processing CreateTablet for tablet f55bf16509404f088a0c2ac6d20c7e67 (DEFAULT_TABLE table=test-table [id=3d998954c7414c4c95063bc92073045b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:00:11.189545 16432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55bf16509404f088a0c2ac6d20c7e67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:00:11.192414 16390 tablet_bootstrap.cc:492] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b: Bootstrap starting.
I20260927 15:00:11.193332 16390 tablet_bootstrap.cc:654] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b: Neither blocks nor log segments found. Creating new log.
I20260927 15:00:11.194118 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:11.187631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33955 (local address 127.0.0.1:59298)
0927 15:00:11.187760 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:11.187776 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:11.187863 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:11.188922 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:11.188926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:11.188945 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:11.189078 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:11.189086 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:11.190629 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:11.190633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:11.192074 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:11.192086 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:11.193553 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:11.193556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:11.193675 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:11.193746 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 15:00:11.193828 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 15:00:11.193927 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":65}
I20260927 15:00:11.194129 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:11.188094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41517 (local address 127.0.0.1:57818)
0927 15:00:11.188362 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:11.188376 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:11.188465 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:11.189204 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:11.189207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:11.189225 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:11.189328 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:11.189334 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:11.190765 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:11.190768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:11.192017 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:11.192029 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:11.193687 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:11.193691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:11.193783 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:11.193806 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:00:11.193881 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 15:00:11.193962 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":3,"thread_start_us":109,"threads_started":1}
I20260927 15:00:11.194159 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:11.188407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57818 (local address 127.0.98.3:41517)
0927 15:00:11.188734 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:11.188745 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:11.188769 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:00:11.188845 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:11.188851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:11.189044 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:11.189473 (+ 429us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:11.189663 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:11.190658 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:11.192141 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:11.193585 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:11.193687 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:11.193821 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:11.193914 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 15:00:11.194028 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":131,"threads_started":1}
I20260927 15:00:11.195007 16390 tablet_bootstrap.cc:492] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b: No bootstrap required, opened a new log
I20260927 15:00:11.195019 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:11.187817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59298 (local address 127.0.98.2:33955)
0927 15:00:11.188376 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:11.188390 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:11.188416 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:00:11.188516 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:11.188522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:11.188770 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:11.188926 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:11.189173 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:11.190522 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:11.192362 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:11.193309 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:11.193429 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:11.193602 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:11.194749 (+ 1147us) server_negotiation.cc:299] Negotiation successful
0927 15:00:11.194894 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":188,"threads_started":1}
I20260927 15:00:11.195122 16390 ts_tablet_manager.cc:1397] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 15:00:11.195628 16390 raft_consensus.cc:348] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b093d925204400bbc7da11cd8629238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33955 } } peers { permanent_uuid: "52a91d287a1e40c58f1f7f7951c20648" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41517 } } peers { permanent_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46197 } }
I20260927 15:00:11.195757 16390 raft_consensus.cc:374] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:00:11.195794 16390 raft_consensus.cc:729] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e934c9fbd86496fa240f8cbfce7c05b, State: Initialized, Role: FOLLOWER
I20260927 15:00:11.195925 16390 consensus_queue.cc:260] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b093d925204400bbc7da11cd8629238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33955 } } peers { permanent_uuid: "52a91d287a1e40c58f1f7f7951c20648" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41517 } } peers { permanent_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46197 } }
I20260927 15:00:11.196197 16390 ts_tablet_manager.cc:1428] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:00:11.197456 16702 tablet_service.cc:1467] Processing CreateTablet for tablet f55bf16509404f088a0c2ac6d20c7e67 (DEFAULT_TABLE table=test-table [id=3d998954c7414c4c95063bc92073045b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:00:11.197744 16570 tablet_service.cc:1467] Processing CreateTablet for tablet f55bf16509404f088a0c2ac6d20c7e67 (DEFAULT_TABLE table=test-table [id=3d998954c7414c4c95063bc92073045b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:00:11.198479 16702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55bf16509404f088a0c2ac6d20c7e67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:00:11.198767 16570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55bf16509404f088a0c2ac6d20c7e67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:00:11.206648 16781 tablet_bootstrap.cc:492] T f55bf16509404f088a0c2ac6d20c7e67 P 4b093d925204400bbc7da11cd8629238: Bootstrap starting.
I20260927 15:00:11.207425 16782 tablet_bootstrap.cc:492] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648: Bootstrap starting.
I20260927 15:00:11.209702 16782 tablet_bootstrap.cc:654] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648: Neither blocks nor log segments found. Creating new log.
I20260927 15:00:11.209702 16781 tablet_bootstrap.cc:654] T f55bf16509404f088a0c2ac6d20c7e67 P 4b093d925204400bbc7da11cd8629238: Neither blocks nor log segments found. Creating new log.
I20260927 15:00:11.210740 16781 log.cc:824] T f55bf16509404f088a0c2ac6d20c7e67 P 4b093d925204400bbc7da11cd8629238: Log is configured to *not* fsync() on all Append() calls
I20260927 15:00:11.210875 16782 log.cc:824] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648: Log is configured to *not* fsync() on all Append() calls
I20260927 15:00:11.213156 16782 tablet_bootstrap.cc:492] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648: No bootstrap required, opened a new log
I20260927 15:00:11.213156 16781 tablet_bootstrap.cc:492] T f55bf16509404f088a0c2ac6d20c7e67 P 4b093d925204400bbc7da11cd8629238: No bootstrap required, opened a new log
I20260927 15:00:11.213397 16782 ts_tablet_manager.cc:1397] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:00:11.213397 16781 ts_tablet_manager.cc:1397] T f55bf16509404f088a0c2ac6d20c7e67 P 4b093d925204400bbc7da11cd8629238: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 15:00:11.217650 16782 raft_consensus.cc:348] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b093d925204400bbc7da11cd8629238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33955 } } peers { permanent_uuid: "52a91d287a1e40c58f1f7f7951c20648" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41517 } } peers { permanent_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46197 } }
I20260927 15:00:11.217650 16781 raft_consensus.cc:348] T f55bf16509404f088a0c2ac6d20c7e67 P 4b093d925204400bbc7da11cd8629238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b093d925204400bbc7da11cd8629238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33955 } } peers { permanent_uuid: "52a91d287a1e40c58f1f7f7951c20648" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41517 } } peers { permanent_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46197 } }
I20260927 15:00:11.218020 16782 raft_consensus.cc:374] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:00:11.218020 16781 raft_consensus.cc:374] T f55bf16509404f088a0c2ac6d20c7e67 P 4b093d925204400bbc7da11cd8629238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:00:11.218079 16781 raft_consensus.cc:729] T f55bf16509404f088a0c2ac6d20c7e67 P 4b093d925204400bbc7da11cd8629238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b093d925204400bbc7da11cd8629238, State: Initialized, Role: FOLLOWER
I20260927 15:00:11.218079 16782 raft_consensus.cc:729] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a91d287a1e40c58f1f7f7951c20648, State: Initialized, Role: FOLLOWER
I20260927 15:00:11.218622 16781 consensus_queue.cc:260] T f55bf16509404f088a0c2ac6d20c7e67 P 4b093d925204400bbc7da11cd8629238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b093d925204400bbc7da11cd8629238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33955 } } peers { permanent_uuid: "52a91d287a1e40c58f1f7f7951c20648" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41517 } } peers { permanent_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46197 } }
I20260927 15:00:11.218622 16782 consensus_queue.cc:260] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b093d925204400bbc7da11cd8629238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33955 } } peers { permanent_uuid: "52a91d287a1e40c58f1f7f7951c20648" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41517 } } peers { permanent_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46197 } }
I20260927 15:00:11.219471 16768 heartbeater.cc:503] Master 127.0.98.62:34767 was elected leader, sending a full tablet report...
I20260927 15:00:11.219696 16636 heartbeater.cc:503] Master 127.0.98.62:34767 was elected leader, sending a full tablet report...
I20260927 15:00:11.219908 16782 ts_tablet_manager.cc:1428] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 15:00:11.219908 16781 ts_tablet_manager.cc:1428] T f55bf16509404f088a0c2ac6d20c7e67 P 4b093d925204400bbc7da11cd8629238: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260927 15:00:11.223030 16499 tablet.cc:2369] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:00:11.382073 16769 tablet.cc:2369] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:00:11.415548 16786 raft_consensus.cc:482] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:00:11.415767 16786 raft_consensus.cc:504] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b093d925204400bbc7da11cd8629238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33955 } } peers { permanent_uuid: "52a91d287a1e40c58f1f7f7951c20648" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41517 } } peers { permanent_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46197 } }
I20260927 15:00:11.416889 16786 leader_election.cc:288] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b093d925204400bbc7da11cd8629238 (127.0.98.2:33955), 2e934c9fbd86496fa240f8cbfce7c05b (127.0.98.1:46197)
I20260927 15:00:11.420816 16770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:11.417158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33955 (local address 127.0.98.3:55209)
0927 15:00:11.417271 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:11.417286 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:11.417383 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:11.417675 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:11.417677 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:11.417697 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:11.417830 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:11.417837 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:11.418669 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:11.418673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:11.419643 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:11.419653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:11.420360 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:11.420364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:11.420470 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:11.420494 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:00:11.420572 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 15:00:11.420663 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":44,"spinlock_wait_cycles":6912}
I20260927 15:00:11.420814 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:11.417321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55209 (local address 127.0.98.2:33955)
0927 15:00:11.417437 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:11.417441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:11.417456 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:00:11.417516 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:11.417519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:11.417572 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:11.417657 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:11.417932 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:11.418554 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:11.419792 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:11.420241 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:11.420325 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:11.420381 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:11.420594 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 15:00:11.420683 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:00:11.421020 16788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:11.417158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46197 (local address 127.0.98.3:60117)
0927 15:00:11.417627 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:11.417641 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:11.417754 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:11.418052 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:11.418055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:11.418073 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:11.418192 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:11.418198 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:11.419001 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:11.419004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:11.419822 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:11.419831 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:11.420606 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:11.420609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:11.420710 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:11.420732 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:00:11.420806 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 15:00:11.420888 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":124,"spinlock_wait_cycles":768,"thread_start_us":123,"threads_started":1}
I20260927 15:00:11.421051 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:11.417258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60117 (local address 127.0.98.1:46197)
0927 15:00:11.417598 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:11.417602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:11.417719 (+ 117us) server_negotiation.cc:407] Connection header received
0927 15:00:11.417852 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:11.417855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:11.417926 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:11.418021 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:11.418288 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:11.418891 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:11.419954 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:11.420493 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:11.420596 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:11.420756 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:11.420824 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 15:00:11.420925 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":120,"threads_started":1}
I20260927 15:00:11.421521 16452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55bf16509404f088a0c2ac6d20c7e67" candidate_uuid: "52a91d287a1e40c58f1f7f7951c20648" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" is_pre_election: true
I20260927 15:00:11.421561 16590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55bf16509404f088a0c2ac6d20c7e67" candidate_uuid: "52a91d287a1e40c58f1f7f7951c20648" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b093d925204400bbc7da11cd8629238" is_pre_election: true
I20260927 15:00:11.421825 16590 raft_consensus.cc:2454] T f55bf16509404f088a0c2ac6d20c7e67 P 4b093d925204400bbc7da11cd8629238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52a91d287a1e40c58f1f7f7951c20648 in term 0.
I20260927 15:00:11.421825 16452 raft_consensus.cc:2454] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52a91d287a1e40c58f1f7f7951c20648 in term 0.
I20260927 15:00:11.422299 16655 leader_election.cc:302] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e934c9fbd86496fa240f8cbfce7c05b, 52a91d287a1e40c58f1f7f7951c20648; no voters:
I20260927 15:00:11.422610 16786 raft_consensus.cc:2781] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:00:11.422694 16786 raft_consensus.cc:482] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:00:11.422727 16786 raft_consensus.cc:3037] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:00:11.423818 16786 raft_consensus.cc:504] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b093d925204400bbc7da11cd8629238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33955 } } peers { permanent_uuid: "52a91d287a1e40c58f1f7f7951c20648" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41517 } } peers { permanent_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46197 } }
I20260927 15:00:11.424192 16786 leader_election.cc:288] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648 [CANDIDATE]: Term 1 election: Requested vote from peers 4b093d925204400bbc7da11cd8629238 (127.0.98.2:33955), 2e934c9fbd86496fa240f8cbfce7c05b (127.0.98.1:46197)
I20260927 15:00:11.424590 16452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55bf16509404f088a0c2ac6d20c7e67" candidate_uuid: "52a91d287a1e40c58f1f7f7951c20648" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e934c9fbd86496fa240f8cbfce7c05b"
I20260927 15:00:11.424594 16590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55bf16509404f088a0c2ac6d20c7e67" candidate_uuid: "52a91d287a1e40c58f1f7f7951c20648" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b093d925204400bbc7da11cd8629238"
I20260927 15:00:11.424716 16452 raft_consensus.cc:3037] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:00:11.424724 16590 raft_consensus.cc:3037] T f55bf16509404f088a0c2ac6d20c7e67 P 4b093d925204400bbc7da11cd8629238 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:00:11.425710 16452 raft_consensus.cc:2454] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52a91d287a1e40c58f1f7f7951c20648 in term 1.
I20260927 15:00:11.425773 16590 raft_consensus.cc:2454] T f55bf16509404f088a0c2ac6d20c7e67 P 4b093d925204400bbc7da11cd8629238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52a91d287a1e40c58f1f7f7951c20648 in term 1.
I20260927 15:00:11.426090 16655 leader_election.cc:302] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e934c9fbd86496fa240f8cbfce7c05b, 52a91d287a1e40c58f1f7f7951c20648; no voters:
I20260927 15:00:11.426281 16786 raft_consensus.cc:2781] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:00:11.426512 16786 raft_consensus.cc:686] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648 [term 1 LEADER]: Becoming Leader. State: Replica: 52a91d287a1e40c58f1f7f7951c20648, State: Running, Role: LEADER
I20260927 15:00:11.426927 16786 consensus_queue.cc:237] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b093d925204400bbc7da11cd8629238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33955 } } peers { permanent_uuid: "52a91d287a1e40c58f1f7f7951c20648" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41517 } } peers { permanent_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46197 } }
I20260927 15:00:11.430301 16310 catalog_manager.cc:5495] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52a91d287a1e40c58f1f7f7951c20648 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "52a91d287a1e40c58f1f7f7951c20648" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b093d925204400bbc7da11cd8629238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52a91d287a1e40c58f1f7f7951c20648" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46197 } health_report { overall_health: UNKNOWN } } }
W20260927 15:00:11.434041 16637 tablet.cc:2369] T f55bf16509404f088a0c2ac6d20c7e67 P 4b093d925204400bbc7da11cd8629238: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:00:11.492472 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:11.488943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57832 (local address 127.0.98.3:41517)
0927 15:00:11.489066 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:11.489069 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:11.489092 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:00:11.489183 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:11.489185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:11.489241 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:11.489327 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:00:11.489515 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:11.490043 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:11.490897 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:11.491163 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:11.491250 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:11.491583 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:00:11.492063 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:00:11.492160 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:11.492285 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 15:00:11.492350 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:00:11.499040 16452 raft_consensus.cc:1261] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [term 1 FOLLOWER]: Refusing update from remote peer 52a91d287a1e40c58f1f7f7951c20648: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:00:11.499045 16590 raft_consensus.cc:1261] T f55bf16509404f088a0c2ac6d20c7e67 P 4b093d925204400bbc7da11cd8629238 [term 1 FOLLOWER]: Refusing update from remote peer 52a91d287a1e40c58f1f7f7951c20648: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:00:11.499850 16789 consensus_queue.cc:1035] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b093d925204400bbc7da11cd8629238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:00:11.500128 16786 consensus_queue.cc:1035] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:00:11.506186 16792 mvcc.cc:205] Tried to move back new op lower bound from 7333974882291134464 to 7333974882008031232. Current Snapshot: MvccSnapshot[applied={T|T < 7333974882291134464}]
I20260927 15:00:11.512458 16795 mvcc.cc:205] Tried to move back new op lower bound from 7333974882291134464 to 7333974882008031232. Current Snapshot: MvccSnapshot[applied={T|T < 7333974882291134464}]
I20260927 15:00:11.518923 16309 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:00:11.519115 16309 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:00:11.521714 16309 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a: Sending DeleteTablet for 3 replicas of tablet f55bf16509404f088a0c2ac6d20c7e67
I20260927 15:00:11.522559 16432 tablet_service.cc:1514] Processing DeleteTablet for tablet f55bf16509404f088a0c2ac6d20c7e67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:00:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33656
I20260927 15:00:11.522547 16570 tablet_service.cc:1514] Processing DeleteTablet for tablet f55bf16509404f088a0c2ac6d20c7e67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:00:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59298
I20260927 15:00:11.522648 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 16348
I20260927 15:00:11.523253 16702 tablet_service.cc:1514] Processing DeleteTablet for tablet f55bf16509404f088a0c2ac6d20c7e67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:00:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57818
I20260927 15:00:11.523674 16802 tablet_replica.cc:330] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648: stopping tablet replica
I20260927 15:00:11.524016 16802 raft_consensus.cc:2229] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:00:11.524423 16802 raft_consensus.cc:2258] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:00:11.525316 16801 tablet_replica.cc:330] T f55bf16509404f088a0c2ac6d20c7e67 P 4b093d925204400bbc7da11cd8629238: stopping tablet replica
I20260927 15:00:11.525683 16801 raft_consensus.cc:2229] T f55bf16509404f088a0c2ac6d20c7e67 P 4b093d925204400bbc7da11cd8629238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:00:11.526010 16801 raft_consensus.cc:2258] T f55bf16509404f088a0c2ac6d20c7e67 P 4b093d925204400bbc7da11cd8629238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:00:11.526094 16802 ts_tablet_manager.cc:1905] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:00:11.527717 16801 ts_tablet_manager.cc:1905] T f55bf16509404f088a0c2ac6d20c7e67 P 4b093d925204400bbc7da11cd8629238: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:00:11.528999 16802 ts_tablet_manager.cc:1918] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:00:11.529105 16802 log.cc:1196] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/wal/wals/f55bf16509404f088a0c2ac6d20c7e67
I20260927 15:00:11.529448 16802 ts_tablet_manager.cc:1939] T f55bf16509404f088a0c2ac6d20c7e67 P 52a91d287a1e40c58f1f7f7951c20648: Deleting consensus metadata
I20260927 15:00:11.530531 16801 ts_tablet_manager.cc:1918] T f55bf16509404f088a0c2ac6d20c7e67 P 4b093d925204400bbc7da11cd8629238: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260927 15:00:11.530580 16294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46197 (error 108)
I20260927 15:00:11.530699 16296 catalog_manager.cc:4841] TS 52a91d287a1e40c58f1f7f7951c20648 (127.0.98.3:41517): tablet f55bf16509404f088a0c2ac6d20c7e67 (table test-table [id=3d998954c7414c4c95063bc92073045b]) successfully deleted
I20260927 15:00:11.530740 16801 log.cc:1196] T f55bf16509404f088a0c2ac6d20c7e67 P 4b093d925204400bbc7da11cd8629238: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/wal/wals/f55bf16509404f088a0c2ac6d20c7e67
I20260927 15:00:11.531268 16801 ts_tablet_manager.cc:1939] T f55bf16509404f088a0c2ac6d20c7e67 P 4b093d925204400bbc7da11cd8629238: Deleting consensus metadata
W20260927 15:00:11.531862 16778 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:00:11.531583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46197 (local address 127.0.0.1:33672)
0927 15:00:11.531714 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:11.531778 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46197: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 15:00:11.531993 16294 catalog_manager.cc:4584] TS 2e934c9fbd86496fa240f8cbfce7c05b (127.0.98.1:46197): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f55bf16509404f088a0c2ac6d20c7e67: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46197: connect: Connection reset by peer (error 104)
I20260927 15:00:11.532343 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 16503
I20260927 15:00:11.533128 16295 catalog_manager.cc:4841] TS 4b093d925204400bbc7da11cd8629238 (127.0.98.2:33955): tablet f55bf16509404f088a0c2ac6d20c7e67 (table test-table [id=3d998954c7414c4c95063bc92073045b]) successfully deleted
I20260927 15:00:11.538226 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 16639
I20260927 15:00:11.543583 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 16277
I20260927 15:00:31.549894 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34767
--webserver_interface=127.0.98.62
--webserver_port=39615
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:00:31.671474 16804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:31.671770 16804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:31.671813 16804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:31.676048 16804 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 15:00:31.676115 16804 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 15:00:31.676131 16804 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 15:00:31.676143 16804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:00:31.676162 16804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:31.676179 16804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:31.676199 16804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:00:31.676229 16804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:00:31.682003 16804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:31.683161 16804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:31.684188 16804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:00:31.693169 16809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:31.693169 16810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:31.693173 16812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:31.693224 16804 server_base.cc:1034] running on GCE node
I20260927 15:00:31.694084 16804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:31.695353 16804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:31.696532 16804 hybrid_clock.cc:648] HybridClock initialized: now 1790521231696509 us; error 39 us; skew 500 ppm
Sep 27 15:00:31 dist-test-slave-pz0h krb5kdc[15729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:00:31.700232 16804 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:00:31.701261 16804 webserver.cc:460] Webserver started at http://127.0.98.62:39615/ using document root <none> and password file <none>
I20260927 15:00:31.701836 16804 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:31.701920 16804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:31.706144 16804 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:00:31.708617 16819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:31.710121 16804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:00:31.710253 16804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "2b08eee532aa45d88003af67633d057a"
format_stamp: "Formatted at 2026-09-27 14:59:48 on dist-test-slave-pz0h"
server_key: "99e5367e5fedfb1a4eeabaab4fb95e20"
server_key_iv: "7e3be957b1ce1e1cbf40e419ba2cd936"
server_key_version: "encryptionkey@0"
I20260927 15:00:31.710878 16804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:31.723943 16804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:31.726784 16804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:31.726994 16804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:31.733755 16804 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34767
I20260927 15:00:31.734014 16871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34767 every 8 connection(s)
I20260927 15:00:31.734859 16804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:00:31.737008 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 16804
I20260927 15:00:31.737426 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46197
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34767
--webserver_port=37049
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:00:31.739410 16872 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:00:31.741324 16872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a: Bootstrap starting.
I20260927 15:00:31.759310 16872 log.cc:824] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a: Log is configured to *not* fsync() on all Append() calls
I20260927 15:00:31.771766 16872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:00:31.772171 16872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a: Bootstrap complete.
I20260927 15:00:31.776170 16872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b08eee532aa45d88003af67633d057a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } }
I20260927 15:00:31.776634 16872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b08eee532aa45d88003af67633d057a, State: Initialized, Role: FOLLOWER
I20260927 15:00:31.777170 16872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b08eee532aa45d88003af67633d057a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } }
I20260927 15:00:31.777304 16872 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:00:31.777374 16872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:00:31.777470 16872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [term 2 FOLLOWER]: Advancing to term 3
I20260927 15:00:31.779646 16872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b08eee532aa45d88003af67633d057a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } }
I20260927 15:00:31.780063 16872 leader_election.cc:302] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b08eee532aa45d88003af67633d057a; no voters:
I20260927 15:00:31.780453 16872 leader_election.cc:288] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 15:00:31.780550 16876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [term 3 FOLLOWER]: Leader election won for term 3
I20260927 15:00:31.780747 16876 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [term 3 LEADER]: Becoming Leader. State: Replica: 2b08eee532aa45d88003af67633d057a, State: Running, Role: LEADER
I20260927 15:00:31.781083 16876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b08eee532aa45d88003af67633d057a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } }
I20260927 15:00:31.781895 16872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:00:31.782588 16877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2b08eee532aa45d88003af67633d057a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b08eee532aa45d88003af67633d057a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } } }
I20260927 15:00:31.782742 16877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [sys.catalog]: This master's current role is: LEADER
I20260927 15:00:31.783486 16878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b08eee532aa45d88003af67633d057a. Latest consensus state: current_term: 3 leader_uuid: "2b08eee532aa45d88003af67633d057a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b08eee532aa45d88003af67633d057a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34767 } } }
I20260927 15:00:31.783655 16878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a [sys.catalog]: This master's current role is: LEADER
I20260927 15:00:31.783865 16885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:00:31.789343 16885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:00:31.790158 16885 catalog_manager.cc:1259] Loaded cluster ID: e0a207cee683435db0e36621f7e4cac1
I20260927 15:00:31.790241 16885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:00:31.792913 16885 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:00:31.793969 16885 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a: Loaded TSK: 0
I20260927 15:00:31.794083 16885 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2b08eee532aa45d88003af67633d057a: Loaded TSK: 1
F20260927 15:00:31.794919 16885 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc8916589ad google::LogMessage::Fail() at ??:0
@ 0x7fc89165b88a google::LogMessage::SendToLog() at ??:0
@ 0x7fc8916584de google::LogMessage::Flush() at ??:0
@ 0x7fc891659a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc89431c18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fc894314f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fc89bfbd7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fc89bf9799c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fc89bf98178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fc89bfd9df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fc89bff882b std::function<>::operator()() at ??:0
@ 0x7fc89bf97b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fc89bf98f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc89bf95579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fc89bfd9640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc8938aa3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc8938aac83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fc8938ac3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc89389b512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc894edf6db start_thread at ??:0
@ 0x7fc89015661f clone at ??:0
W20260927 15:00:31.888077 16874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:31.888491 16874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:31.888581 16874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:31.894500 16874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:31.894589 16874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:31.894707 16874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:00:31.901347 16874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46197
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37049
--tserver_master_addrs=127.0.98.62:34767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:31.902526 16874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:31.903625 16874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:00:31.913411 16919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:31.913411 16922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:31.913444 16920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:31.914142 16874 server_base.cc:1034] running on GCE node
I20260927 15:00:31.914564 16874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:31.915299 16874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:31.916463 16874 hybrid_clock.cc:648] HybridClock initialized: now 1790521231916432 us; error 54 us; skew 500 ppm
Sep 27 15:00:31 dist-test-slave-pz0h krb5kdc[15729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:00:31.919286 16874 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 15:00:31.920341 16874 webserver.cc:460] Webserver started at http://127.0.98.1:37049/ using document root <none> and password file <none>
I20260927 15:00:31.920920 16874 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:31.920984 16874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:31.925076 16874 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:00:31.927493 16929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:31.928912 16874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:00:31.929044 16874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "2e934c9fbd86496fa240f8cbfce7c05b"
format_stamp: "Formatted at 2026-09-27 14:59:49 on dist-test-slave-pz0h"
server_key: "4473e166550f139ffcc405c8b25b5b8c"
server_key_iv: "b4412227f296a39a1110fe733686bf39"
server_key_version: "encryptionkey@0"
I20260927 15:00:31.929633 16874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:31.950062 16874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:31.953317 16874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:31.953518 16874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:31.954144 16874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:00:31.955384 16936 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 15:00:31.959611 16874 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 15:00:31.959687 16874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 15:00:31.959754 16874 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 15:00:31.961598 16874 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 15:00:31.961676 16874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 15:00:31.961984 16936 tablet_bootstrap.cc:492] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b: Bootstrap starting.
I20260927 15:00:31.972067 16874 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46197
I20260927 15:00:31.972445 17043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46197 every 8 connection(s)
I20260927 15:00:31.973423 16874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:00:31.974781 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 16874
I20260927 15:00:31.975296 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33955
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34767
--webserver_port=33521
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:00:31.984205 16936 log.cc:824] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:00:31.994666 16936 tablet_bootstrap.cc:492] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:00:31.995240 16936 tablet_bootstrap.cc:492] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b: Bootstrap complete.
I20260927 15:00:31.996948 16936 ts_tablet_manager.cc:1397] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b: Time spent bootstrapping tablet: real 0.035s user 0.028s sys 0.004s
I20260927 15:00:32.000814 16936 raft_consensus.cc:348] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b093d925204400bbc7da11cd8629238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33955 } } peers { permanent_uuid: "52a91d287a1e40c58f1f7f7951c20648" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41517 } } peers { permanent_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46197 } }
I20260927 15:00:32.001502 16936 raft_consensus.cc:729] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e934c9fbd86496fa240f8cbfce7c05b, State: Initialized, Role: FOLLOWER
I20260927 15:00:32.002172 16936 consensus_queue.cc:260] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b093d925204400bbc7da11cd8629238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33955 } } peers { permanent_uuid: "52a91d287a1e40c58f1f7f7951c20648" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41517 } } peers { permanent_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46197 } }
I20260927 15:00:32.003446 16936 ts_tablet_manager.cc:1428] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260927 15:00:32.028594 17047 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:00:31.976411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:38391)
0927 15:00:31.977852 (+ 1441us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:31.977933 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:31.979155 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:32.027526 (+ 48371us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1185,"thread_start_us":189,"threads_started":1}
W20260927 15:00:32.029209 17044 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34767 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34767: Client connection negotiation failed: client connection to 127.0.98.62:34767: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 15:00:32.030028 17047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:32.029709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:39353)
0927 15:00:32.029868 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:32.029953 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 15:00:32.031085 17047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:32.030776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:41721)
0927 15:00:32.030950 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:32.031014 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260927 15:00:32.031378 17044 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:00:32.100063 17046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:32.100387 17046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:32.100435 17046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:32.104763 17046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:32.104838 17046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:32.104909 17046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:00:32.110633 17046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33521
--tserver_master_addrs=127.0.98.62:34767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:32.111828 17046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:32.112876 17046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:00:32.122968 17057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:32.122969 17054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:32.122968 17055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:32.123183 17046 server_base.cc:1034] running on GCE node
I20260927 15:00:32.123862 17046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:32.124607 17046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:32.125788 17046 hybrid_clock.cc:648] HybridClock initialized: now 1790521232125758 us; error 48 us; skew 500 ppm
Sep 27 15:00:32 dist-test-slave-pz0h krb5kdc[15729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:00:32.129139 17046 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 15:00:32.130760 17046 webserver.cc:460] Webserver started at http://127.0.98.2:33521/ using document root <none> and password file <none>
I20260927 15:00:32.131459 17046 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:32.131543 17046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:32.135907 17046 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 15:00:32.138622 17064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:32.140254 17046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:00:32.140422 17046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "4b093d925204400bbc7da11cd8629238"
format_stamp: "Formatted at 2026-09-27 14:59:49 on dist-test-slave-pz0h"
server_key: "256fd6339acfeba332ca4f967668b99f"
server_key_iv: "8e0ba0bdc18e5bc67593adf95b6f54b4"
server_key_version: "encryptionkey@0"
I20260927 15:00:32.141062 17046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:32.157544 17046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:32.160938 17046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:32.161152 17046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:32.161789 17046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:00:32.162686 17046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:00:32.162760 17046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:32.162861 17046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:00:32.162891 17046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:32.172049 17046 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33955
I20260927 15:00:32.172086 17177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33955 every 8 connection(s)
I20260927 15:00:32.173146 17046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:00:32.175216 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 17046
I20260927 15:00:32.175628 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41517
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34767
--webserver_port=44667
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:00:32.177778 17180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:32.175613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:53169)
0927 15:00:32.176186 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:32.176417 (+ 231us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":170,"threads_started":1}
W20260927 15:00:32.178290 17178 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34767 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34767: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
I20260927 15:00:32.179060 17180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:32.178755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:50735)
0927 15:00:32.178933 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:32.178991 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260927 15:00:32.179718 17180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:32.179518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:43787)
0927 15:00:32.179615 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:32.179663 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 15:00:32.179920 17178 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:00:32.291862 17181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:32.292140 17181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:32.292188 17181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:32.296401 17181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:32.296483 17181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:32.296545 17181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:00:32.302207 17181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44667
--tserver_master_addrs=127.0.98.62:34767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:32.303431 17181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:32.304512 17181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:00:32.314936 17187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:32.314983 17181 server_base.cc:1034] running on GCE node
W20260927 15:00:32.314955 17186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:32.314936 17189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:32.315593 17181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:32.316326 17181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:32.317533 17181 hybrid_clock.cc:648] HybridClock initialized: now 1790521232317508 us; error 44 us; skew 500 ppm
Sep 27 15:00:32 dist-test-slave-pz0h krb5kdc[15729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:00:32.320950 17181 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 15:00:32.322024 17181 webserver.cc:460] Webserver started at http://127.0.98.3:44667/ using document root <none> and password file <none>
I20260927 15:00:32.322628 17181 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:32.322702 17181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:32.327540 17181 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:00:32.330348 17196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:32.332031 17181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:00:32.332180 17181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "52a91d287a1e40c58f1f7f7951c20648"
format_stamp: "Formatted at 2026-09-27 14:59:49 on dist-test-slave-pz0h"
server_key: "50583edddd9df8eb9bc45e9d6f27decd"
server_key_iv: "3993a5bf504ef6eabc3724bf399d889c"
server_key_version: "encryptionkey@0"
I20260927 15:00:32.332832 17181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:32.357846 17181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:32.361325 17181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:32.361523 17181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:32.362170 17181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:00:32.363238 17181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:00:32.363291 17181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:32.363345 17181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:00:32.363376 17181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:32.373898 17181 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41517
I20260927 15:00:32.373955 17309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41517 every 8 connection(s)
I20260927 15:00:32.375012 17181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:00:32.379415 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:32.377288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:40143)
0927 15:00:32.377884 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:32.378033 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":159,"threads_started":1}
W20260927 15:00:32.379933 17310 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34767 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34767: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
I20260927 15:00:32.380606 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:32.380371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:35833)
0927 15:00:32.380489 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:32.380542 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 15:00:32.381246 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:32.381053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:36827)
0927 15:00:32.381152 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:32.381195 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 15:00:32.381424 17310 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 15:00:32.383150 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 17181
I20260927 15:00:33.032718 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:33.032231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:42935)
0927 15:00:33.032539 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:33.032604 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":90,"threads_started":1}
I20260927 15:00:33.180974 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:33.180500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:34491)
0927 15:00:33.180782 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:33.180839 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":85,"threads_started":1}
I20260927 15:00:33.382663 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:33.382140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:40891)
0927 15:00:33.382459 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:33.382525 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":78,"threads_started":1}
I20260927 15:00:33.525594 17317 raft_consensus.cc:482] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:00:33.525904 17317 raft_consensus.cc:504] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b093d925204400bbc7da11cd8629238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33955 } } peers { permanent_uuid: "52a91d287a1e40c58f1f7f7951c20648" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41517 } } peers { permanent_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46197 } }
I20260927 15:00:33.527665 17317 leader_election.cc:288] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b093d925204400bbc7da11cd8629238 (127.0.98.2:33955), 52a91d287a1e40c58f1f7f7951c20648 (127.0.98.3:41517)
Sep 27 15:00:33 dist-test-slave-pz0h krb5kdc[15729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 27 15:00:33 dist-test-slave-pz0h krb5kdc[15729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260927 15:00:33.541580 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:33.527974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33955 (local address 127.0.98.1:52277)
0927 15:00:33.528987 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:33.529000 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:33.529118 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:33.530300 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:33.530303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:33.530725 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:00:33.531219 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:33.531236 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:33.532500 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:33.532503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:33.533581 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:33.533591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:33.534267 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:33.534271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:33.534486 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:33.535141 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:00:33.535158 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:00:33.536651 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:00:33.538735 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:00:33.538742 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:00:33.538757 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:00:33.539052 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:00:33.539371 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:00:33.539373 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:00:33.539375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:00:33.539551 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:00:33.540352 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:00:33.540360 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:00:33.540740 (+ 380us) client_negotiation.cc:769] Sending connection context
0927 15:00:33.540957 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 15:00:33.541359 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"mutex_wait_us":630,"spinlock_wait_cycles":50048,"thread_start_us":86,"threads_started":1}
I20260927 15:00:33.541633 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:33.528153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52277 (local address 127.0.98.2:33955)
0927 15:00:33.528538 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:33.528560 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:33.529104 (+ 544us) server_negotiation.cc:407] Connection header received
0927 15:00:33.529786 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:33.529791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:33.530091 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:33.530327 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:33.531378 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:33.532357 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:33.533687 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:33.534149 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:33.534299 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:33.536879 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:33.536886 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:33.536901 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:00:33.536933 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:33.538606 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:33.539160 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:33.539167 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:33.539171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:33.539290 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:33.539638 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:33.539641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:33.539643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:33.540215 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:33.540354 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:33.541121 (+ 767us) server_negotiation.cc:299] Negotiation successful
0927 15:00:33.541438 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":114,"threads_started":1}
I20260927 15:00:33.541633 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:33.528110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49555 (local address 127.0.98.3:41517)
0927 15:00:33.528593 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:33.528619 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:33.528663 (+ 44us) server_negotiation.cc:407] Connection header received
0927 15:00:33.529418 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:33.529425 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:33.529795 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:33.530090 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:33.531378 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:33.532376 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:33.533688 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:33.534188 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:33.534384 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:33.537225 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:33.537232 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:33.537247 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:00:33.537282 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:33.538926 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:33.539364 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:33.539369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:33.539374 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:33.539492 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:33.539797 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:33.539800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:33.539802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:33.540215 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:33.540351 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:33.541121 (+ 770us) server_negotiation.cc:299] Negotiation successful
0927 15:00:33.541438 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":147,"threads_started":1}
I20260927 15:00:33.541579 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:33.527972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41517 (local address 127.0.98.1:49555)
0927 15:00:33.528199 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:33.528216 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:33.528332 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:33.530170 (+ 1838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:33.530176 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:33.530720 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:00:33.531219 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:33.531236 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:33.532502 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:33.532505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:33.533582 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:33.533591 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:33.534320 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:33.534324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:33.534490 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:33.535195 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:00:33.535207 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:00:33.537048 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:00:33.539026 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:00:33.539029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:00:33.539032 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:00:33.539245 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:00:33.539582 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:00:33.539585 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:00:33.539587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:00:33.539689 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:00:33.540352 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:00:33.540360 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:00:33.540740 (+ 380us) client_negotiation.cc:769] Sending connection context
0927 15:00:33.540957 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 15:00:33.541424 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":369,"spinlock_wait_cycles":13312}
W20260927 15:00:33.544381 16931 leader_election.cc:341] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4b093d925204400bbc7da11cd8629238 (127.0.98.2:33955): Not found: Tablet not found: f55bf16509404f088a0c2ac6d20c7e67
W20260927 15:00:33.544540 16932 leader_election.cc:341] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 52a91d287a1e40c58f1f7f7951c20648 (127.0.98.3:41517): Not found: Tablet not found: f55bf16509404f088a0c2ac6d20c7e67
I20260927 15:00:33.544648 16932 leader_election.cc:302] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e934c9fbd86496fa240f8cbfce7c05b; no voters: 4b093d925204400bbc7da11cd8629238, 52a91d287a1e40c58f1f7f7951c20648
I20260927 15:00:33.544946 17317 raft_consensus.cc:2726] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:00:34.033938 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:34.033600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:46347)
0927 15:00:34.033780 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:34.033848 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 15:00:34.182548 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:34.182016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:54057)
0927 15:00:34.182330 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:34.182420 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":87,"threads_started":1}
I20260927 15:00:34.383889 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:34.383460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:50485)
0927 15:00:34.383725 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:34.383788 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":80,"threads_started":1}
I20260927 15:00:35.035172 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:35.034671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:57333)
0927 15:00:35.034998 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:35.035066 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":108,"threads_started":1}
I20260927 15:00:35.175866 17324 raft_consensus.cc:482] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:00:35.176003 17324 raft_consensus.cc:504] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b093d925204400bbc7da11cd8629238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33955 } } peers { permanent_uuid: "52a91d287a1e40c58f1f7f7951c20648" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41517 } } peers { permanent_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46197 } }
I20260927 15:00:35.176532 17324 leader_election.cc:288] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b093d925204400bbc7da11cd8629238 (127.0.98.2:33955), 52a91d287a1e40c58f1f7f7951c20648 (127.0.98.3:41517)
W20260927 15:00:35.177424 16931 leader_election.cc:341] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4b093d925204400bbc7da11cd8629238 (127.0.98.2:33955): Not found: Tablet not found: f55bf16509404f088a0c2ac6d20c7e67
W20260927 15:00:35.177534 16932 leader_election.cc:341] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 52a91d287a1e40c58f1f7f7951c20648 (127.0.98.3:41517): Not found: Tablet not found: f55bf16509404f088a0c2ac6d20c7e67
I20260927 15:00:35.177615 16932 leader_election.cc:302] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e934c9fbd86496fa240f8cbfce7c05b; no voters: 4b093d925204400bbc7da11cd8629238, 52a91d287a1e40c58f1f7f7951c20648
I20260927 15:00:35.177771 17324 raft_consensus.cc:2726] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:00:35.183815 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:35.183392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:35939)
0927 15:00:35.183633 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:35.183699 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":76,"threads_started":1}
I20260927 15:00:35.385248 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:35.384702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:39169)
0927 15:00:35.385011 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:35.385100 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":79,"threads_started":1}
I20260927 15:00:36.036459 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:36.035994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:40239)
0927 15:00:36.036287 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:36.036358 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":81,"threads_started":1}
I20260927 15:00:36.184998 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:36.184586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:47445)
0927 15:00:36.184844 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:36.184900 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":77,"threads_started":1}
I20260927 15:00:36.386716 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:36.386261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:39233)
0927 15:00:36.386537 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:36.386603 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":78,"threads_started":1}
I20260927 15:00:37.037889 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:37.037409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:55575)
0927 15:00:37.037728 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:37.037790 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":71,"threads_started":1}
I20260927 15:00:37.186370 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:37.185926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:49173)
0927 15:00:37.186195 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:37.186270 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":79,"threads_started":1}
I20260927 15:00:37.388212 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:37.387734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:54895)
0927 15:00:37.388046 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:37.388115 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":94,"threads_started":1}
I20260927 15:00:38.039180 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:38.038703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:55039)
0927 15:00:38.039011 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:38.039078 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":124,"threads_started":1}
I20260927 15:00:38.187624 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:38.187173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:47393)
0927 15:00:38.187461 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:38.187521 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":79,"threads_started":1}
I20260927 15:00:38.389755 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:38.389262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:52949)
0927 15:00:38.389583 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:38.389651 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":86,"threads_started":1}
I20260927 15:00:38.548138 17336 raft_consensus.cc:482] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:00:38.548305 17336 raft_consensus.cc:504] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b093d925204400bbc7da11cd8629238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33955 } } peers { permanent_uuid: "52a91d287a1e40c58f1f7f7951c20648" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41517 } } peers { permanent_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46197 } }
I20260927 15:00:38.548774 17336 leader_election.cc:288] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b093d925204400bbc7da11cd8629238 (127.0.98.2:33955), 52a91d287a1e40c58f1f7f7951c20648 (127.0.98.3:41517)
W20260927 15:00:38.549628 16931 leader_election.cc:341] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4b093d925204400bbc7da11cd8629238 (127.0.98.2:33955): Not found: Tablet not found: f55bf16509404f088a0c2ac6d20c7e67
W20260927 15:00:38.549749 16932 leader_election.cc:341] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 52a91d287a1e40c58f1f7f7951c20648 (127.0.98.3:41517): Not found: Tablet not found: f55bf16509404f088a0c2ac6d20c7e67
I20260927 15:00:38.549853 16932 leader_election.cc:302] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e934c9fbd86496fa240f8cbfce7c05b; no voters: 4b093d925204400bbc7da11cd8629238, 52a91d287a1e40c58f1f7f7951c20648
I20260927 15:00:38.550038 17336 raft_consensus.cc:2726] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:00:39.040529 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:39.040061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:35709)
0927 15:00:39.040365 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:39.040431 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":81,"threads_started":1}
I20260927 15:00:39.189186 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:39.188665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:41301)
0927 15:00:39.189006 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:39.189087 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":92,"threads_started":1}
I20260927 15:00:39.391160 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:39.390646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:54153)
0927 15:00:39.390994 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:39.391063 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":79,"threads_started":1}
I20260927 15:00:40.041929 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:40.041405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:52727)
0927 15:00:40.041764 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:40.041828 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":116,"threads_started":1}
I20260927 15:00:40.190536 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:40.190053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:57733)
0927 15:00:40.190337 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:40.190411 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":90,"threads_started":1}
I20260927 15:00:40.392367 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:40.391932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:42821)
0927 15:00:40.392208 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:40.392270 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":85,"threads_started":1}
I20260927 15:00:41.043143 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:41.042698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:57951)
0927 15:00:41.042976 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:41.043037 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":114,"threads_started":1}
I20260927 15:00:41.191867 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:41.191380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:57427)
0927 15:00:41.191705 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:41.191768 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":118,"threads_started":1}
I20260927 15:00:41.331507 17345 raft_consensus.cc:482] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:00:41.331668 17345 raft_consensus.cc:504] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b093d925204400bbc7da11cd8629238" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33955 } } peers { permanent_uuid: "52a91d287a1e40c58f1f7f7951c20648" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41517 } } peers { permanent_uuid: "2e934c9fbd86496fa240f8cbfce7c05b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46197 } }
I20260927 15:00:41.332139 17345 leader_election.cc:288] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b093d925204400bbc7da11cd8629238 (127.0.98.2:33955), 52a91d287a1e40c58f1f7f7951c20648 (127.0.98.3:41517)
W20260927 15:00:41.332902 16931 leader_election.cc:341] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4b093d925204400bbc7da11cd8629238 (127.0.98.2:33955): Not found: Tablet not found: f55bf16509404f088a0c2ac6d20c7e67
W20260927 15:00:41.333007 16932 leader_election.cc:341] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 52a91d287a1e40c58f1f7f7951c20648 (127.0.98.3:41517): Not found: Tablet not found: f55bf16509404f088a0c2ac6d20c7e67
I20260927 15:00:41.333115 16932 leader_election.cc:302] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e934c9fbd86496fa240f8cbfce7c05b; no voters: 4b093d925204400bbc7da11cd8629238, 52a91d287a1e40c58f1f7f7951c20648
I20260927 15:00:41.333247 17345 raft_consensus.cc:2726] T f55bf16509404f088a0c2ac6d20c7e67 P 2e934c9fbd86496fa240f8cbfce7c05b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:00:41.393549 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:41.393140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:45191)
0927 15:00:41.393387 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:41.393443 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":78,"threads_started":1}
I20260927 15:00:42.044564 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:42.044116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:54565)
0927 15:00:42.044385 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:42.044449 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":81,"threads_started":1}
I20260927 15:00:42.193248 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:42.192759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:34395)
0927 15:00:42.193072 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:42.193139 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":79,"threads_started":1}
I20260927 15:00:42.394977 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:42.394441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:38689)
0927 15:00:42.394753 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:42.394869 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":84,"threads_started":1}
I20260927 15:00:43.045799 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:43.045340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:58697)
0927 15:00:43.045634 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:43.045698 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":76,"threads_started":1}
I20260927 15:00:43.194675 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:43.194204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:46853)
0927 15:00:43.194499 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:43.194564 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":90,"threads_started":1}
I20260927 15:00:43.396395 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:43.395947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:45465)
0927 15:00:43.396232 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:43.396295 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":92,"threads_started":1}
I20260927 15:00:44.047124 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:44.046678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:53913)
0927 15:00:44.046966 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:44.047026 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":79,"threads_started":1}
I20260927 15:00:44.196055 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:44.195577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:56427)
0927 15:00:44.195870 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:44.195947 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":80,"threads_started":1}
I20260927 15:00:44.397850 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:44.397354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:45627)
0927 15:00:44.397668 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:44.397736 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":83,"threads_started":1}
I20260927 15:00:45.048450 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:45.047994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:47893)
0927 15:00:45.048270 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:45.048333 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":78,"threads_started":1}
I20260927 15:00:45.197306 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:45.196878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:39709)
0927 15:00:45.197148 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:45.197208 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":91,"threads_started":1}
I20260927 15:00:45.399333 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:45.398782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:51505)
0927 15:00:45.399151 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:45.399214 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":99,"threads_started":1}
I20260927 15:00:46.049796 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:46.049329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:53851)
0927 15:00:46.049632 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:46.049695 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":86,"threads_started":1}
I20260927 15:00:46.198992 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:46.198472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:37281)
0927 15:00:46.198783 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:46.198889 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":97,"threads_started":1}
I20260927 15:00:46.400548 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:46.400099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.3:54391)
0927 15:00:46.400385 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:46.400448 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":84,"threads_started":1}
I20260927 15:00:47.051401 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:47.050920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.1:40039)
0927 15:00:47.051217 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:47.051301 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":77,"threads_started":1}
I20260927 15:00:47.200287 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:47.199876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34767 (local address 127.0.98.2:34987)
0927 15:00:47.200132 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:47.200189 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":82,"threads_started":1}
I20260927 15:00:47.385102 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 16874
I20260927 15:00:47.390951 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 17046
I20260927 15:00:47.396716 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 17181
2026-09-27T15:00:47Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59638 ms)
[----------] 30 tests from SecurityITest (243149 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:00:47 dist-test-slave-pz0h krb5kdc[17366](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:00:47 dist-test-slave-pz0h krb5kdc[17366](info): set up 2 sockets
Sep 27 15:00:47 dist-test-slave-pz0h krb5kdc[17366](info): commencing operation
krb5kdc: starting...
W20260927 15:00:48.358618 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 15:00:48 dist-test-slave-pz0h krb5kdc[17366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:00:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:00:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:00:48.480063 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36699
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36699
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:00:48.596027 17379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:48.596344 17379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:48.596390 17379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:48.600670 17379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:00:48.600756 17379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:48.600775 17379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:48.600791 17379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:00:48.600806 17379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:00:48.606531 17379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:48.607726 17379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:48.608757 17379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:00:48.617942 17379 server_base.cc:1034] running on GCE node
W20260927 15:00:48.617883 17385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:48.617892 17384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:48.617883 17387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:48.618886 17379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:48.620167 17379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:48.621428 17379 hybrid_clock.cc:648] HybridClock initialized: now 1790521248621375 us; error 58 us; skew 500 ppm
Sep 27 15:00:48 dist-test-slave-pz0h krb5kdc[17366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:00:48.625141 17379 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:00:48.626204 17379 webserver.cc:460] Webserver started at http://127.0.98.62:33313/ using document root <none> and password file <none>
I20260927 15:00:48.626785 17379 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:48.626910 17379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:48.627125 17379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:00:48.629314 17379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "ddd520c6b7874909a0da85fcad3a7c8b"
format_stamp: "Formatted at 2026-09-27 15:00:48 on dist-test-slave-pz0h"
server_key: "13b117f136806e313cbac16ff7fc7c45"
server_key_iv: "db6d161726f433a3a5ab088a3768a3a9"
server_key_version: "encryptionkey@0"
I20260927 15:00:48.629917 17379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "ddd520c6b7874909a0da85fcad3a7c8b"
format_stamp: "Formatted at 2026-09-27 15:00:48 on dist-test-slave-pz0h"
server_key: "13b117f136806e313cbac16ff7fc7c45"
server_key_iv: "db6d161726f433a3a5ab088a3768a3a9"
server_key_version: "encryptionkey@0"
I20260927 15:00:48.634066 17379 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 15:00:48.636957 17394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:48.638513 17379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 15:00:48.638667 17379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "ddd520c6b7874909a0da85fcad3a7c8b"
format_stamp: "Formatted at 2026-09-27 15:00:48 on dist-test-slave-pz0h"
server_key: "13b117f136806e313cbac16ff7fc7c45"
server_key_iv: "db6d161726f433a3a5ab088a3768a3a9"
server_key_version: "encryptionkey@0"
I20260927 15:00:48.638839 17379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:48.651194 17379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:48.654201 17379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:48.654556 17379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:48.661765 17379 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36699
I20260927 15:00:48.661810 17446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36699 every 8 connection(s)
I20260927 15:00:48.662783 17379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:00:48.665674 17447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:00:48.666931 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 17379
I20260927 15:00:48.667063 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 15:00:48.667327 392 external_mini_cluster.cc:1428] Setting key 399b3ddb1caa441b1690eb45ddd6566f
I20260927 15:00:48.673133 17447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddd520c6b7874909a0da85fcad3a7c8b: Bootstrap starting.
I20260927 15:00:48.675922 17447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddd520c6b7874909a0da85fcad3a7c8b: Neither blocks nor log segments found. Creating new log.
I20260927 15:00:48.676792 17447 log.cc:824] T 00000000000000000000000000000000 P ddd520c6b7874909a0da85fcad3a7c8b: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:00:48 dist-test-slave-pz0h krb5kdc[17366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:00:48.679103 17447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddd520c6b7874909a0da85fcad3a7c8b: No bootstrap required, opened a new log
I20260927 15:00:48.682338 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:48.668650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53902 (local address 127.0.98.62:36699)
0927 15:00:48.669100 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:48.669119 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:48.669164 (+ 45us) server_negotiation.cc:407] Connection header received
0927 15:00:48.669997 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:48.670022 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:48.670490 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:48.670925 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:48.671644 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:48.673111 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:48.674059 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:48.674762 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:48.674964 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:48.677701 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:48.677707 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:48.677724 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:00:48.677763 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:48.679471 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:48.679923 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:48.679929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:48.679935 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:48.680059 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:48.680324 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:48.680327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:48.680328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:48.680764 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:48.680975 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:48.681242 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 15:00:48.681562 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":120,"threads_started":1}
I20260927 15:00:48.682935 17447 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddd520c6b7874909a0da85fcad3a7c8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddd520c6b7874909a0da85fcad3a7c8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36699 } }
I20260927 15:00:48.683178 17447 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddd520c6b7874909a0da85fcad3a7c8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:00:48.683269 17447 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddd520c6b7874909a0da85fcad3a7c8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddd520c6b7874909a0da85fcad3a7c8b, State: Initialized, Role: FOLLOWER
I20260927 15:00:48.684854 17447 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddd520c6b7874909a0da85fcad3a7c8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddd520c6b7874909a0da85fcad3a7c8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36699 } }
I20260927 15:00:48.685019 17447 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddd520c6b7874909a0da85fcad3a7c8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:00:48.685084 17447 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddd520c6b7874909a0da85fcad3a7c8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:00:48.685184 17447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddd520c6b7874909a0da85fcad3a7c8b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:00:48.686311 17447 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddd520c6b7874909a0da85fcad3a7c8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddd520c6b7874909a0da85fcad3a7c8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36699 } }
I20260927 15:00:48.686748 17447 leader_election.cc:302] T 00000000000000000000000000000000 P ddd520c6b7874909a0da85fcad3a7c8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddd520c6b7874909a0da85fcad3a7c8b; no voters:
I20260927 15:00:48.687150 17447 leader_election.cc:288] T 00000000000000000000000000000000 P ddd520c6b7874909a0da85fcad3a7c8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:00:48.687251 17452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddd520c6b7874909a0da85fcad3a7c8b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:00:48.687489 17452 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddd520c6b7874909a0da85fcad3a7c8b [term 1 LEADER]: Becoming Leader. State: Replica: ddd520c6b7874909a0da85fcad3a7c8b, State: Running, Role: LEADER
I20260927 15:00:48.687801 17452 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddd520c6b7874909a0da85fcad3a7c8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddd520c6b7874909a0da85fcad3a7c8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36699 } }
I20260927 15:00:48.688422 17447 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddd520c6b7874909a0da85fcad3a7c8b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:00:48.689474 17454 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddd520c6b7874909a0da85fcad3a7c8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddd520c6b7874909a0da85fcad3a7c8b. Latest consensus state: current_term: 1 leader_uuid: "ddd520c6b7874909a0da85fcad3a7c8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddd520c6b7874909a0da85fcad3a7c8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36699 } } }
I20260927 15:00:48.689477 17453 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddd520c6b7874909a0da85fcad3a7c8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddd520c6b7874909a0da85fcad3a7c8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddd520c6b7874909a0da85fcad3a7c8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36699 } } }
I20260927 15:00:48.689664 17454 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddd520c6b7874909a0da85fcad3a7c8b [sys.catalog]: This master's current role is: LEADER
I20260927 15:00:48.689664 17453 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddd520c6b7874909a0da85fcad3a7c8b [sys.catalog]: This master's current role is: LEADER
I20260927 15:00:48.690107 17461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:00:48.693863 17461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:00:48.700034 17461 catalog_manager.cc:1347] Generated new cluster ID: 62871a8245e94e33a3e82d4ace405b06
I20260927 15:00:48.700191 17461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:00:48.720733 17461 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:00:48.721903 17461 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:00:48.733450 17461 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ddd520c6b7874909a0da85fcad3a7c8b: Generated new TSK 0
I20260927 15:00:48.734233 17461 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 15:00:48.769652 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36699
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:00:48.891153 17473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:48.891469 17473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:48.891513 17473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:48.895816 17473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:48.895893 17473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:48.895963 17473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:00:48.901608 17473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:48.902828 17473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:48.903895 17473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:00:48.913851 17481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:48.913856 17479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:48.913854 17478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:48.914079 17473 server_base.cc:1034] running on GCE node
I20260927 15:00:48.914682 17473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:48.915438 17473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:48.916591 17473 hybrid_clock.cc:648] HybridClock initialized: now 1790521248916564 us; error 47 us; skew 500 ppm
Sep 27 15:00:48 dist-test-slave-pz0h krb5kdc[17366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:00:48.919970 17473 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 15:00:48.920997 17473 webserver.cc:460] Webserver started at http://127.0.98.1:41195/ using document root <none> and password file <none>
I20260927 15:00:48.921593 17473 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:48.921663 17473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:48.921878 17473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:00:48.924099 17473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-0/data/instance:
uuid: "18c4f8586e764e78a74e5d243e63b437"
format_stamp: "Formatted at 2026-09-27 15:00:48 on dist-test-slave-pz0h"
server_key: "64799359d91f7475a9737cb35c34b8de"
server_key_iv: "e883abe328f1147034962ffd5f7862d4"
server_key_version: "encryptionkey@0"
I20260927 15:00:48.924692 17473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "18c4f8586e764e78a74e5d243e63b437"
format_stamp: "Formatted at 2026-09-27 15:00:48 on dist-test-slave-pz0h"
server_key: "64799359d91f7475a9737cb35c34b8de"
server_key_iv: "e883abe328f1147034962ffd5f7862d4"
server_key_version: "encryptionkey@0"
I20260927 15:00:48.929147 17473 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:00:48.932098 17488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:48.933481 17473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:00:48.933629 17473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "18c4f8586e764e78a74e5d243e63b437"
format_stamp: "Formatted at 2026-09-27 15:00:48 on dist-test-slave-pz0h"
server_key: "64799359d91f7475a9737cb35c34b8de"
server_key_iv: "e883abe328f1147034962ffd5f7862d4"
server_key_version: "encryptionkey@0"
I20260927 15:00:48.933756 17473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:48.950533 17473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:48.955220 17473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:48.955499 17473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:48.956179 17473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:00:48.957129 17473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:00:48.957180 17473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:48.957235 17473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:00:48.957264 17473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:48.966288 17473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32869
I20260927 15:00:48.966310 17601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32869 every 8 connection(s)
I20260927 15:00:48.967375 17473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:00:48.976918 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 17473
Sep 27 15:00:48 dist-test-slave-pz0h krb5kdc[17366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:00:48.977056 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:00:48.977366 392 external_mini_cluster.cc:1428] Setting key 4e53b973f3355e5f83595699761e92f4
I20260927 15:00:48.981570 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:48.969295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56721 (local address 127.0.98.62:36699)
0927 15:00:48.969475 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:48.969479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:48.970550 (+ 1071us) server_negotiation.cc:407] Connection header received
0927 15:00:48.971525 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:48.971528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:48.971591 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:48.971668 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:48.972971 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:48.973521 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:48.974703 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:48.974991 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:48.975092 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:48.977804 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:48.977809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:48.977812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:00:48.977847 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:48.979294 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:48.979930 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:48.979933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:48.979935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:48.980017 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:48.980395 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:48.980397 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:48.980398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:48.980642 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:48.980738 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:48.981258 (+ 520us) server_negotiation.cc:299] Negotiation successful
0927 15:00:48.981403 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 15:00:48.982376 17604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:48.969652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36699 (local address 127.0.98.1:56721)
0927 15:00:48.970369 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:48.970418 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:48.971301 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:48.971873 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:48.971881 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:48.972288 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:00:48.972803 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:48.972818 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:48.973652 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:48.973655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:48.974571 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:48.974582 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:48.975155 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:48.975159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:48.975332 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:48.975961 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:00:48.975979 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:00:48.977582 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:00:48.979422 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:00:48.979429 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:00:48.979448 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:00:48.979816 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:00:48.980097 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:00:48.980100 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:00:48.980102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:00:48.980306 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:00:48.980739 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:00:48.980745 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:00:48.981028 (+ 283us) client_negotiation.cc:769] Sending connection context
0927 15:00:48.981199 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 15:00:48.981488 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":122,"threads_started":1}
I20260927 15:00:48.983806 17602 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36699
I20260927 15:00:48.984068 17602 heartbeater.cc:464] Registering TS with master...
I20260927 15:00:48.984612 17602 heartbeater.cc:511] Master 127.0.98.62:36699 requested a full tablet report, sending...
I20260927 15:00:48.986544 17411 ts_manager.cc:194] Registered new tserver with Master: 18c4f8586e764e78a74e5d243e63b437 (127.0.98.1:32869)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 15:00:48.987957 17411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 15:00:49.006358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36699
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:00:49.121831 17607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:49.122143 17607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:49.122208 17607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:49.126904 17607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:49.126988 17607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:49.127076 17607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:00:49.132771 17607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:49.133931 17607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:49.135039 17607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:00:49.145529 17615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:49.145529 17612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:49.145529 17613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:49.145764 17607 server_base.cc:1034] running on GCE node
I20260927 15:00:49.146334 17607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:49.147082 17607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:49.148285 17607 hybrid_clock.cc:648] HybridClock initialized: now 1790521249148265 us; error 33 us; skew 500 ppm
Sep 27 15:00:49 dist-test-slave-pz0h krb5kdc[17366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:00:49.151319 17607 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 15:00:49.152354 17607 webserver.cc:460] Webserver started at http://127.0.98.2:34355/ using document root <none> and password file <none>
I20260927 15:00:49.152937 17607 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:49.153007 17607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:49.153206 17607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:00:49.155524 17607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-1/data/instance:
uuid: "bb6be22959514293b8d778f81e373630"
format_stamp: "Formatted at 2026-09-27 15:00:49 on dist-test-slave-pz0h"
server_key: "78e190d808920a7ff2eb2c0ae513c96d"
server_key_iv: "d2667dce6b4768dcbb99fa35518e34ff"
server_key_version: "encryptionkey@0"
I20260927 15:00:49.156181 17607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bb6be22959514293b8d778f81e373630"
format_stamp: "Formatted at 2026-09-27 15:00:49 on dist-test-slave-pz0h"
server_key: "78e190d808920a7ff2eb2c0ae513c96d"
server_key_iv: "d2667dce6b4768dcbb99fa35518e34ff"
server_key_version: "encryptionkey@0"
I20260927 15:00:49.160478 17607 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:00:49.163468 17622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:49.164803 17607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:00:49.164939 17607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "bb6be22959514293b8d778f81e373630"
format_stamp: "Formatted at 2026-09-27 15:00:49 on dist-test-slave-pz0h"
server_key: "78e190d808920a7ff2eb2c0ae513c96d"
server_key_iv: "d2667dce6b4768dcbb99fa35518e34ff"
server_key_version: "encryptionkey@0"
I20260927 15:00:49.165062 17607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:49.190536 17607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:49.194213 17607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:49.194429 17607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:49.195099 17607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:00:49.195960 17607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:00:49.196008 17607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:49.196051 17607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:00:49.196071 17607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:49.204612 17607 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38335
I20260927 15:00:49.204633 17735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38335 every 8 connection(s)
I20260927 15:00:49.205596 17607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:00:49.213637 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 17607
I20260927 15:00:49.213752 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:00:49.214047 392 external_mini_cluster.cc:1428] Setting key 52cbbaf222b82055d8c10620cf39e347
Sep 27 15:00:49 dist-test-slave-pz0h krb5kdc[17366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:00:49.220850 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:49.207603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35059 (local address 127.0.98.62:36699)
0927 15:00:49.207739 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:49.207742 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:49.208673 (+ 931us) server_negotiation.cc:407] Connection header received
0927 15:00:49.209605 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:49.209608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:49.209668 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:49.209740 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:49.211008 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.211539 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.212897 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.213125 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.213220 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:49.216236 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:49.216240 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:49.216242 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:00:49.216389 (+ 147us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:49.218338 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:49.219035 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:49.219038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:49.219040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:49.219134 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:49.219550 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:49.219553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:49.219554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:49.219759 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:49.219857 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:49.220588 (+ 731us) server_negotiation.cc:299] Negotiation successful
0927 15:00:49.220721 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:00:49.222409 17738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:49.207984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36699 (local address 127.0.98.2:35059)
0927 15:00:49.208498 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:49.208549 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:49.209392 (+ 843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:49.209923 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:49.209931 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:49.210307 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:00:49.210861 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:49.210878 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.211759 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.211762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:49.212740 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:49.212751 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.213293 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.213297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:49.213518 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:49.214218 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:00:49.214237 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:00:49.216055 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:00:49.218492 (+ 2437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:00:49.218499 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:00:49.218518 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:00:49.218915 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:00:49.219241 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:00:49.219244 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:00:49.219246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:00:49.219442 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:00:49.219871 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:00:49.219880 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:00:49.220249 (+ 369us) client_negotiation.cc:769] Sending connection context
0927 15:00:49.220492 (+ 243us) client_negotiation.cc:240] Negotiation successful
0927 15:00:49.220903 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":130,"threads_started":1}
I20260927 15:00:49.223790 17736 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36699
I20260927 15:00:49.224049 17736 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 15:00:49.224596 17736 heartbeater.cc:511] Master 127.0.98.62:36699 requested a full tablet report, sending...
I20260927 15:00:49.225800 17411 ts_manager.cc:194] Registered new tserver with Master: bb6be22959514293b8d778f81e373630 (127.0.98.2:38335)
I20260927 15:00:49.226409 17411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 15:00:49.241477 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36699
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:00:49.361091 17741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:49.361583 17741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:49.361663 17741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:49.367240 17741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:49.367363 17741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:49.367457 17741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:00:49.374245 17741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:49.375509 17741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:49.376580 17741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:00:49.386674 17747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:49.386686 17746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:49.386674 17749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:49.386955 17741 server_base.cc:1034] running on GCE node
I20260927 15:00:49.387517 17741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:49.388233 17741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:49.389426 17741 hybrid_clock.cc:648] HybridClock initialized: now 1790521249389408 us; error 33 us; skew 500 ppm
Sep 27 15:00:49 dist-test-slave-pz0h krb5kdc[17366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:00:49.393508 17741 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 15:00:49.394768 17741 webserver.cc:460] Webserver started at http://127.0.98.3:42683/ using document root <none> and password file <none>
I20260927 15:00:49.395443 17741 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:49.395514 17741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:49.395699 17741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:00:49.397912 17741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-2/data/instance:
uuid: "f116ad0b6838472eafcea49849ccc625"
format_stamp: "Formatted at 2026-09-27 15:00:49 on dist-test-slave-pz0h"
server_key: "16bd6ec83c0180a30e5541280ffd5571"
server_key_iv: "a5a94a0db32d14f4b0de7a2f13342065"
server_key_version: "encryptionkey@0"
I20260927 15:00:49.398505 17741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f116ad0b6838472eafcea49849ccc625"
format_stamp: "Formatted at 2026-09-27 15:00:49 on dist-test-slave-pz0h"
server_key: "16bd6ec83c0180a30e5541280ffd5571"
server_key_iv: "a5a94a0db32d14f4b0de7a2f13342065"
server_key_version: "encryptionkey@0"
I20260927 15:00:49.402560 17741 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260927 15:00:49.405407 17756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:49.406917 17741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:00:49.407079 17741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "f116ad0b6838472eafcea49849ccc625"
format_stamp: "Formatted at 2026-09-27 15:00:49 on dist-test-slave-pz0h"
server_key: "16bd6ec83c0180a30e5541280ffd5571"
server_key_iv: "a5a94a0db32d14f4b0de7a2f13342065"
server_key_version: "encryptionkey@0"
I20260927 15:00:49.407202 17741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:49.425176 17741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:49.428630 17741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:49.428833 17741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:49.429459 17741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:00:49.430341 17741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:00:49.430392 17741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:49.430444 17741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:00:49.430473 17741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:49.440605 17741 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41623
I20260927 15:00:49.440634 17869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41623 every 8 connection(s)
I20260927 15:00:49.441532 17741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:00:49.448858 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 17741
I20260927 15:00:49.449023 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-2/wal/instance
I20260927 15:00:49.449324 392 external_mini_cluster.cc:1428] Setting key 3c9744e2162baa89247f6b0225d77f5b
Sep 27 15:00:49 dist-test-slave-pz0h krb5kdc[17366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:00:49.458132 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:49.443475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59985 (local address 127.0.98.62:36699)
0927 15:00:49.443643 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:49.443647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:49.444916 (+ 1269us) server_negotiation.cc:407] Connection header received
0927 15:00:49.445870 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:49.445874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:49.445926 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:49.446011 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:49.447326 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.447888 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.449226 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.449545 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.449668 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:49.453222 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:49.453227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:49.453230 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:00:49.453266 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:49.455037 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:49.455729 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:49.455733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:49.455735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:49.455834 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:49.456189 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:49.456193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:49.456195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:49.456631 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:49.456778 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:49.457645 (+ 867us) server_negotiation.cc:299] Negotiation successful
0927 15:00:49.457948 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 15:00:49.459131 17872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:49.443995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36699 (local address 127.0.98.3:59985)
0927 15:00:49.444761 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:49.444808 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:49.445637 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:49.446184 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:49.446191 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:49.446615 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:00:49.447158 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:49.447173 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.448047 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.448050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:49.449049 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:49.449063 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.449753 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.449758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:49.449976 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:49.451011 (+ 1035us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:00:49.451039 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:00:49.453000 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:00:49.455220 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:00:49.455228 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:00:49.455247 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:00:49.455603 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:00:49.455928 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:00:49.455930 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:00:49.455931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:00:49.456091 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:00:49.456880 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:00:49.456890 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:00:49.457307 (+ 417us) client_negotiation.cc:769] Sending connection context
0927 15:00:49.457568 (+ 261us) client_negotiation.cc:240] Negotiation successful
0927 15:00:49.458131 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":140,"threads_started":1}
I20260927 15:00:49.460613 17870 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36699
I20260927 15:00:49.460894 17870 heartbeater.cc:464] Registering TS with master...
I20260927 15:00:49.461532 17870 heartbeater.cc:511] Master 127.0.98.62:36699 requested a full tablet report, sending...
I20260927 15:00:49.462845 17411 ts_manager.cc:194] Registered new tserver with Master: f116ad0b6838472eafcea49849ccc625 (127.0.98.3:41623)
I20260927 15:00:49.463562 17411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59985
I20260927 15:00:49.465139 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:00:49.475765 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:49.467877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54608 (local address 127.0.98.62:36699)
0927 15:00:49.468036 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:49.468040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:49.468145 (+ 105us) server_negotiation.cc:407] Connection header received
0927 15:00:49.468273 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:49.468276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:49.468323 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:49.468390 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:49.469174 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.469707 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.470571 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.470846 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.470951 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:49.472472 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:49.472475 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:49.472478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:00:49.472504 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:49.474022 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:49.474457 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:49.474461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:49.474463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:49.474543 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:49.474787 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:49.474822 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:49.474824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:49.475092 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:49.475225 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:49.475473 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 15:00:49.475598 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:00:49.479151 17411 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:00:49.482442 17411 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:00:49.499560 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:49.494300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39392 (local address 127.0.98.3:41623)
0927 15:00:49.494751 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:49.494773 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:49.494820 (+ 47us) server_negotiation.cc:407] Connection header received
0927 15:00:49.494896 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:49.494901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:49.495056 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:49.495208 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:49.495390 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.496474 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.497991 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.498588 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.498690 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:49.498894 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:49.499261 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 15:00:49.499381 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":132,"threads_started":1}
I20260927 15:00:49.499554 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:49.493900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41623 (local address 127.0.0.1:39392)
0927 15:00:49.494539 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:49.494553 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:49.494641 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:49.495173 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:49.495176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:49.495191 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:49.495293 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:49.495300 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.496589 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.496593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:49.497851 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:49.497862 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.498892 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.498896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:49.499024 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:49.499070 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 15:00:49.499216 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 15:00:49.499380 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":72,"thread_start_us":91,"threads_started":1}
I20260927 15:00:49.499930 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:49.493113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32869 (local address 127.0.0.1:59236)
0927 15:00:49.493512 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:49.493568 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:49.493697 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:49.494770 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:49.494776 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:49.494822 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:49.494986 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:49.494994 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.496500 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.496504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:49.498446 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:49.498475 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.499577 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.499581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:49.499681 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:49.499706 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 15:00:49.499759 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 15:00:49.499828 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":176,"thread_start_us":101,"threads_started":1}
I20260927 15:00:49.500171 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:49.493283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38335 (local address 127.0.0.1:43950)
0927 15:00:49.493925 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:49.493937 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:49.494016 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:49.495335 (+ 1319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:49.495338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:49.495355 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:49.495449 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:49.495453 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.496533 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.496536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:49.498446 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:49.498475 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.499771 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.499775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:49.499886 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:49.499912 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 15:00:49.499959 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 15:00:49.500039 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":220,"thread_start_us":116,"threads_started":1}
I20260927 15:00:49.500743 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:49.493998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43950 (local address 127.0.98.2:38335)
0927 15:00:49.494896 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:49.494909 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:49.494930 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:00:49.494999 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:49.495004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:49.495193 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:49.495560 (+ 367us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:49.495683 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.496403 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.498720 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.499629 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.500332 (+ 703us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:49.500488 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:49.500541 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 15:00:49.500622 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":768,"thread_start_us":138,"threads_started":1}
I20260927 15:00:49.501101 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:49.493267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59236 (local address 127.0.98.1:32869)
0927 15:00:49.494249 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:49.494269 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:49.494295 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:00:49.494373 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:49.494378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:49.494601 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:49.494765 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:49.495121 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.496361 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.498707 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.499427 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.500469 (+ 1042us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:49.500623 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:49.500674 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 15:00:49.500950 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":877,"thread_start_us":101,"threads_started":1}
I20260927 15:00:49.503201 17804 tablet_service.cc:1467] Processing CreateTablet for tablet 11d870c002654ca1b4b022067969d33a (DEFAULT_TABLE table=test-table [id=18483340e70c4070b5a2c88b29f062af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:00:49.504271 17804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11d870c002654ca1b4b022067969d33a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:00:49.504002 17536 tablet_service.cc:1467] Processing CreateTablet for tablet 11d870c002654ca1b4b022067969d33a (DEFAULT_TABLE table=test-table [id=18483340e70c4070b5a2c88b29f062af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:00:49.504004 17670 tablet_service.cc:1467] Processing CreateTablet for tablet 11d870c002654ca1b4b022067969d33a (DEFAULT_TABLE table=test-table [id=18483340e70c4070b5a2c88b29f062af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:00:49.505115 17536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11d870c002654ca1b4b022067969d33a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:00:49.505115 17670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11d870c002654ca1b4b022067969d33a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:00:49.511912 17887 tablet_bootstrap.cc:492] T 11d870c002654ca1b4b022067969d33a P f116ad0b6838472eafcea49849ccc625: Bootstrap starting.
I20260927 15:00:49.513247 17889 tablet_bootstrap.cc:492] T 11d870c002654ca1b4b022067969d33a P 18c4f8586e764e78a74e5d243e63b437: Bootstrap starting.
I20260927 15:00:49.513957 17888 tablet_bootstrap.cc:492] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630: Bootstrap starting.
I20260927 15:00:49.514501 17887 tablet_bootstrap.cc:654] T 11d870c002654ca1b4b022067969d33a P f116ad0b6838472eafcea49849ccc625: Neither blocks nor log segments found. Creating new log.
I20260927 15:00:49.515627 17889 tablet_bootstrap.cc:654] T 11d870c002654ca1b4b022067969d33a P 18c4f8586e764e78a74e5d243e63b437: Neither blocks nor log segments found. Creating new log.
I20260927 15:00:49.515779 17888 tablet_bootstrap.cc:654] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630: Neither blocks nor log segments found. Creating new log.
I20260927 15:00:49.515865 17887 log.cc:824] T 11d870c002654ca1b4b022067969d33a P f116ad0b6838472eafcea49849ccc625: Log is configured to *not* fsync() on all Append() calls
I20260927 15:00:49.516477 17889 log.cc:824] T 11d870c002654ca1b4b022067969d33a P 18c4f8586e764e78a74e5d243e63b437: Log is configured to *not* fsync() on all Append() calls
I20260927 15:00:49.516883 17888 log.cc:824] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630: Log is configured to *not* fsync() on all Append() calls
I20260927 15:00:49.518265 17887 tablet_bootstrap.cc:492] T 11d870c002654ca1b4b022067969d33a P f116ad0b6838472eafcea49849ccc625: No bootstrap required, opened a new log
I20260927 15:00:49.518505 17887 ts_tablet_manager.cc:1397] T 11d870c002654ca1b4b022067969d33a P f116ad0b6838472eafcea49849ccc625: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 15:00:49.518849 17888 tablet_bootstrap.cc:492] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630: No bootstrap required, opened a new log
I20260927 15:00:49.519102 17889 tablet_bootstrap.cc:492] T 11d870c002654ca1b4b022067969d33a P 18c4f8586e764e78a74e5d243e63b437: No bootstrap required, opened a new log
I20260927 15:00:49.519222 17888 ts_tablet_manager.cc:1397] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 15:00:49.519307 17889 ts_tablet_manager.cc:1397] T 11d870c002654ca1b4b022067969d33a P 18c4f8586e764e78a74e5d243e63b437: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:00:49.522271 17888 raft_consensus.cc:348] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c4f8586e764e78a74e5d243e63b437" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32869 } } peers { permanent_uuid: "bb6be22959514293b8d778f81e373630" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38335 } } peers { permanent_uuid: "f116ad0b6838472eafcea49849ccc625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260927 15:00:49.522576 17888 raft_consensus.cc:374] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:00:49.522642 17888 raft_consensus.cc:729] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6be22959514293b8d778f81e373630, State: Initialized, Role: FOLLOWER
I20260927 15:00:49.522742 17887 raft_consensus.cc:348] T 11d870c002654ca1b4b022067969d33a P f116ad0b6838472eafcea49849ccc625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c4f8586e764e78a74e5d243e63b437" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32869 } } peers { permanent_uuid: "bb6be22959514293b8d778f81e373630" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38335 } } peers { permanent_uuid: "f116ad0b6838472eafcea49849ccc625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260927 15:00:49.523080 17887 raft_consensus.cc:374] T 11d870c002654ca1b4b022067969d33a P f116ad0b6838472eafcea49849ccc625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:00:49.523151 17887 raft_consensus.cc:729] T 11d870c002654ca1b4b022067969d33a P f116ad0b6838472eafcea49849ccc625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f116ad0b6838472eafcea49849ccc625, State: Initialized, Role: FOLLOWER
I20260927 15:00:49.523180 17888 consensus_queue.cc:260] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c4f8586e764e78a74e5d243e63b437" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32869 } } peers { permanent_uuid: "bb6be22959514293b8d778f81e373630" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38335 } } peers { permanent_uuid: "f116ad0b6838472eafcea49849ccc625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260927 15:00:49.523607 17887 consensus_queue.cc:260] T 11d870c002654ca1b4b022067969d33a P f116ad0b6838472eafcea49849ccc625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c4f8586e764e78a74e5d243e63b437" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32869 } } peers { permanent_uuid: "bb6be22959514293b8d778f81e373630" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38335 } } peers { permanent_uuid: "f116ad0b6838472eafcea49849ccc625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260927 15:00:49.524083 17736 heartbeater.cc:503] Master 127.0.98.62:36699 was elected leader, sending a full tablet report...
I20260927 15:00:49.523948 17889 raft_consensus.cc:348] T 11d870c002654ca1b4b022067969d33a P 18c4f8586e764e78a74e5d243e63b437 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c4f8586e764e78a74e5d243e63b437" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32869 } } peers { permanent_uuid: "bb6be22959514293b8d778f81e373630" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38335 } } peers { permanent_uuid: "f116ad0b6838472eafcea49849ccc625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260927 15:00:49.524223 17889 raft_consensus.cc:374] T 11d870c002654ca1b4b022067969d33a P 18c4f8586e764e78a74e5d243e63b437 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:00:49.524269 17889 raft_consensus.cc:729] T 11d870c002654ca1b4b022067969d33a P 18c4f8586e764e78a74e5d243e63b437 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c4f8586e764e78a74e5d243e63b437, State: Initialized, Role: FOLLOWER
I20260927 15:00:49.524449 17888 ts_tablet_manager.cc:1428] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:00:49.524642 17887 ts_tablet_manager.cc:1428] T 11d870c002654ca1b4b022067969d33a P f116ad0b6838472eafcea49849ccc625: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 15:00:49.525152 17870 heartbeater.cc:503] Master 127.0.98.62:36699 was elected leader, sending a full tablet report...
I20260927 15:00:49.525465 17889 consensus_queue.cc:260] T 11d870c002654ca1b4b022067969d33a P 18c4f8586e764e78a74e5d243e63b437 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c4f8586e764e78a74e5d243e63b437" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32869 } } peers { permanent_uuid: "bb6be22959514293b8d778f81e373630" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38335 } } peers { permanent_uuid: "f116ad0b6838472eafcea49849ccc625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260927 15:00:49.526441 17602 heartbeater.cc:503] Master 127.0.98.62:36699 was elected leader, sending a full tablet report...
I20260927 15:00:49.526499 17889 ts_tablet_manager.cc:1428] T 11d870c002654ca1b4b022067969d33a P 18c4f8586e764e78a74e5d243e63b437: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260927 15:00:49.692552 17871 tablet.cc:2369] T 11d870c002654ca1b4b022067969d33a P f116ad0b6838472eafcea49849ccc625: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:00:49.706926 17737 tablet.cc:2369] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:00:49.718937 17603 tablet.cc:2369] T 11d870c002654ca1b4b022067969d33a P 18c4f8586e764e78a74e5d243e63b437: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:00:49.835355 17893 raft_consensus.cc:482] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:00:49.835574 17893 raft_consensus.cc:504] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c4f8586e764e78a74e5d243e63b437" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32869 } } peers { permanent_uuid: "bb6be22959514293b8d778f81e373630" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38335 } } peers { permanent_uuid: "f116ad0b6838472eafcea49849ccc625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260927 15:00:49.836760 17893 leader_election.cc:288] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18c4f8586e764e78a74e5d243e63b437 (127.0.98.1:32869), f116ad0b6838472eafcea49849ccc625 (127.0.98.3:41623)
I20260927 15:00:49.840767 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:49.837208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52043 (local address 127.0.98.3:41623)
0927 15:00:49.837331 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:49.837336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:49.837468 (+ 132us) server_negotiation.cc:407] Connection header received
0927 15:00:49.837647 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:49.837649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:49.837690 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:49.837745 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:49.838072 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.838573 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.839710 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.840134 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.840231 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:49.840298 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:49.840545 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 15:00:49.840624 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:00:49.840768 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:49.837133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56581 (local address 127.0.98.1:32869)
0927 15:00:49.837262 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:49.837266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:49.837489 (+ 223us) server_negotiation.cc:407] Connection header received
0927 15:00:49.837636 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:49.837639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:49.837683 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:49.837746 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:49.838072 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.838653 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.839721 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.840152 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.840247 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:49.840310 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:49.840554 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 15:00:49.840632 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:00:49.840795 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:49.837084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41623 (local address 127.0.98.2:52043)
0927 15:00:49.837387 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:49.837398 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:49.837534 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:49.837774 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:49.837777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:49.837801 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:49.837943 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:49.837949 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.838685 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.838689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:49.839593 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:49.839603 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.840256 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.840259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:49.840420 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:49.840451 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 15:00:49.840519 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 15:00:49.840625 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":69,"spinlock_wait_cycles":16768,"thread_start_us":64,"threads_started":1}
I20260927 15:00:49.840798 17897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:49.837008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32869 (local address 127.0.98.2:56581)
0927 15:00:49.837348 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:49.837360 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:49.837534 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:49.837771 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:49.837774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:49.837800 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:49.837960 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:49.837965 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.838751 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.838754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:49.839607 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:49.839616 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.840267 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.840271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:49.840443 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:49.840463 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 15:00:49.840527 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 15:00:49.840625 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":101,"spinlock_wait_cycles":34304,"thread_start_us":103,"threads_started":1}
I20260927 15:00:49.841442 17556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11d870c002654ca1b4b022067969d33a" candidate_uuid: "bb6be22959514293b8d778f81e373630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c4f8586e764e78a74e5d243e63b437" is_pre_election: true
I20260927 15:00:49.841442 17824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11d870c002654ca1b4b022067969d33a" candidate_uuid: "bb6be22959514293b8d778f81e373630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f116ad0b6838472eafcea49849ccc625" is_pre_election: true
I20260927 15:00:49.841779 17556 raft_consensus.cc:2454] T 11d870c002654ca1b4b022067969d33a P 18c4f8586e764e78a74e5d243e63b437 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb6be22959514293b8d778f81e373630 in term 0.
I20260927 15:00:49.841779 17824 raft_consensus.cc:2454] T 11d870c002654ca1b4b022067969d33a P f116ad0b6838472eafcea49849ccc625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb6be22959514293b8d778f81e373630 in term 0.
I20260927 15:00:49.842326 17623 leader_election.cc:302] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18c4f8586e764e78a74e5d243e63b437, bb6be22959514293b8d778f81e373630; no voters:
I20260927 15:00:49.842614 17893 raft_consensus.cc:2781] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:00:49.842697 17893 raft_consensus.cc:482] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:00:49.842748 17893 raft_consensus.cc:3037] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:00:49.844048 17893 raft_consensus.cc:504] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c4f8586e764e78a74e5d243e63b437" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32869 } } peers { permanent_uuid: "bb6be22959514293b8d778f81e373630" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38335 } } peers { permanent_uuid: "f116ad0b6838472eafcea49849ccc625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260927 15:00:49.844432 17893 leader_election.cc:288] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630 [CANDIDATE]: Term 1 election: Requested vote from peers 18c4f8586e764e78a74e5d243e63b437 (127.0.98.1:32869), f116ad0b6838472eafcea49849ccc625 (127.0.98.3:41623)
I20260927 15:00:49.844888 17556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11d870c002654ca1b4b022067969d33a" candidate_uuid: "bb6be22959514293b8d778f81e373630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c4f8586e764e78a74e5d243e63b437"
I20260927 15:00:49.845016 17824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11d870c002654ca1b4b022067969d33a" candidate_uuid: "bb6be22959514293b8d778f81e373630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f116ad0b6838472eafcea49849ccc625"
I20260927 15:00:49.845053 17556 raft_consensus.cc:3037] T 11d870c002654ca1b4b022067969d33a P 18c4f8586e764e78a74e5d243e63b437 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:00:49.845148 17824 raft_consensus.cc:3037] T 11d870c002654ca1b4b022067969d33a P f116ad0b6838472eafcea49849ccc625 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:00:49.846421 17824 raft_consensus.cc:2454] T 11d870c002654ca1b4b022067969d33a P f116ad0b6838472eafcea49849ccc625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb6be22959514293b8d778f81e373630 in term 1.
I20260927 15:00:49.846421 17556 raft_consensus.cc:2454] T 11d870c002654ca1b4b022067969d33a P 18c4f8586e764e78a74e5d243e63b437 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb6be22959514293b8d778f81e373630 in term 1.
I20260927 15:00:49.846858 17623 leader_election.cc:302] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18c4f8586e764e78a74e5d243e63b437, bb6be22959514293b8d778f81e373630; no voters:
I20260927 15:00:49.847054 17893 raft_consensus.cc:2781] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:00:49.847287 17893 raft_consensus.cc:686] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630 [term 1 LEADER]: Becoming Leader. State: Replica: bb6be22959514293b8d778f81e373630, State: Running, Role: LEADER
I20260927 15:00:49.847623 17893 consensus_queue.cc:237] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c4f8586e764e78a74e5d243e63b437" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32869 } } peers { permanent_uuid: "bb6be22959514293b8d778f81e373630" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38335 } } peers { permanent_uuid: "f116ad0b6838472eafcea49849ccc625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260927 15:00:49.850880 17411 catalog_manager.cc:5495] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb6be22959514293b8d778f81e373630 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bb6be22959514293b8d778f81e373630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c4f8586e764e78a74e5d243e63b437" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb6be22959514293b8d778f81e373630" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f116ad0b6838472eafcea49849ccc625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } health_report { overall_health: UNKNOWN } } }
I20260927 15:00:49.869257 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:49.865812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43952 (local address 127.0.98.2:38335)
0927 15:00:49.865953 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:49.865958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:49.865974 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:00:49.866071 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:49.866074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:49.866129 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:49.866230 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:00:49.866423 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.866988 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.867859 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:49.868081 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:49.868167 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:49.868429 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:00:49.868892 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:00:49.868982 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:49.869044 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 15:00:49.869103 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:00:49.875841 17824 raft_consensus.cc:1261] T 11d870c002654ca1b4b022067969d33a P f116ad0b6838472eafcea49849ccc625 [term 1 FOLLOWER]: Refusing update from remote peer bb6be22959514293b8d778f81e373630: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:00:49.875841 17556 raft_consensus.cc:1261] T 11d870c002654ca1b4b022067969d33a P 18c4f8586e764e78a74e5d243e63b437 [term 1 FOLLOWER]: Refusing update from remote peer bb6be22959514293b8d778f81e373630: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:00:49.876693 17893 consensus_queue.cc:1035] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c4f8586e764e78a74e5d243e63b437" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:00:49.876943 17898 consensus_queue.cc:1035] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f116ad0b6838472eafcea49849ccc625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:00:49.887084 17902 mvcc.cc:205] Tried to move back new op lower bound from 7333975039482421248 to 7333975039378624512. Current Snapshot: MvccSnapshot[applied={T|T < 7333975039482421248}]
I20260927 15:00:49.893864 17408 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:00:49.894049 17408 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:00:49.896332 17408 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ddd520c6b7874909a0da85fcad3a7c8b: Sending DeleteTablet for 3 replicas of tablet 11d870c002654ca1b4b022067969d33a
I20260927 15:00:49.897208 17536 tablet_service.cc:1514] Processing DeleteTablet for tablet 11d870c002654ca1b4b022067969d33a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:00:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59236
I20260927 15:00:49.897360 17670 tablet_service.cc:1514] Processing DeleteTablet for tablet 11d870c002654ca1b4b022067969d33a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:00:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43950
I20260927 15:00:49.897832 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 17473
I20260927 15:00:49.897795 17804 tablet_service.cc:1514] Processing DeleteTablet for tablet 11d870c002654ca1b4b022067969d33a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:00:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39392
I20260927 15:00:49.898294 17912 tablet_replica.cc:330] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630: stopping tablet replica
I20260927 15:00:49.898520 17913 tablet_replica.cc:330] T 11d870c002654ca1b4b022067969d33a P f116ad0b6838472eafcea49849ccc625: stopping tablet replica
I20260927 15:00:49.898778 17912 raft_consensus.cc:2229] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:00:49.898777 17913 raft_consensus.cc:2229] T 11d870c002654ca1b4b022067969d33a P f116ad0b6838472eafcea49849ccc625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:00:49.899154 17913 raft_consensus.cc:2258] T 11d870c002654ca1b4b022067969d33a P f116ad0b6838472eafcea49849ccc625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:00:49.899289 17912 raft_consensus.cc:2258] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:00:49.900852 17912 ts_tablet_manager.cc:1905] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:00:49.900852 17913 ts_tablet_manager.cc:1905] T 11d870c002654ca1b4b022067969d33a P f116ad0b6838472eafcea49849ccc625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:00:49.903715 17913 ts_tablet_manager.cc:1918] T 11d870c002654ca1b4b022067969d33a P f116ad0b6838472eafcea49849ccc625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:00:49.903715 17912 ts_tablet_manager.cc:1918] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:00:49.903829 17912 log.cc:1196] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-1/wal/wals/11d870c002654ca1b4b022067969d33a
I20260927 15:00:49.903829 17913 log.cc:1196] T 11d870c002654ca1b4b022067969d33a P f116ad0b6838472eafcea49849ccc625: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790521004237705-392-0/minicluster-data/ts-2/wal/wals/11d870c002654ca1b4b022067969d33a
I20260927 15:00:49.904132 17913 ts_tablet_manager.cc:1939] T 11d870c002654ca1b4b022067969d33a P f116ad0b6838472eafcea49849ccc625: Deleting consensus metadata
I20260927 15:00:49.904132 17912 ts_tablet_manager.cc:1939] T 11d870c002654ca1b4b022067969d33a P bb6be22959514293b8d778f81e373630: Deleting consensus metadata
I20260927 15:00:49.905434 17396 catalog_manager.cc:4841] TS bb6be22959514293b8d778f81e373630 (127.0.98.2:38335): tablet 11d870c002654ca1b4b022067969d33a (table test-table [id=18483340e70c4070b5a2c88b29f062af]) successfully deleted
I20260927 15:00:49.905539 17397 catalog_manager.cc:4841] TS f116ad0b6838472eafcea49849ccc625 (127.0.98.3:41623): tablet 11d870c002654ca1b4b022067969d33a (table test-table [id=18483340e70c4070b5a2c88b29f062af]) successfully deleted
I20260927 15:00:49.909423 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 17607
W20260927 15:00:49.909370 17395 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32869 (error 108)
I20260927 15:00:49.911202 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:49.910717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32869 (local address 127.0.0.1:59244)
0927 15:00:49.910939 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:49.911045 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 15:00:49.911419 17395 catalog_manager.cc:4584] TS 18c4f8586e764e78a74e5d243e63b437 (127.0.98.1:32869): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 11d870c002654ca1b4b022067969d33a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32869: connect: Connection refused (error 111)
I20260927 15:00:49.916050 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 17741
I20260927 15:00:49.921330 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 17379
2026-09-27T15:00:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2525 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T15:00:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:00:49Z Disabled control of system clock
I20260927 15:00:49.952124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46555
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:00:50.084619 17918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:50.084920 17918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:50.084957 17918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:50.089125 17918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:00:50.089200 17918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:50.089214 17918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:50.089231 17918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:00:50.089246 17918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:00:50.094702 17918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46555
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:50.095846 17918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:50.097234 17918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:00:50.106482 17918 server_base.cc:1034] running on GCE node
W20260927 15:00:50.106434 17923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:50.106434 17926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:50.106434 17924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:50.107427 17918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:50.108788 17918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:50.110234 17918 hybrid_clock.cc:648] HybridClock initialized: now 1790521250110199 us; error 42 us; skew 500 ppm
I20260927 15:00:50.113085 17918 webserver.cc:460] Webserver started at http://127.0.98.62:44609/ using document root <none> and password file <none>
I20260927 15:00:50.113744 17918 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:50.113823 17918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:50.114027 17918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:00:50.116269 17918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "27c9be341bd340c5b516fbeac9ec5052"
format_stamp: "Formatted at 2026-09-27 15:00:50 on dist-test-slave-pz0h"
server_key: "658a21cb8e599d7b8a74c3f39fc1efac"
server_key_iv: "b6524e8c28fb2519f6f536075f6968c3"
server_key_version: "encryptionkey@0"
I20260927 15:00:50.116884 17918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "27c9be341bd340c5b516fbeac9ec5052"
format_stamp: "Formatted at 2026-09-27 15:00:50 on dist-test-slave-pz0h"
server_key: "658a21cb8e599d7b8a74c3f39fc1efac"
server_key_iv: "b6524e8c28fb2519f6f536075f6968c3"
server_key_version: "encryptionkey@0"
I20260927 15:00:50.121238 17918 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:00:50.124128 17932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:50.125547 17918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:00:50.125713 17918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "27c9be341bd340c5b516fbeac9ec5052"
format_stamp: "Formatted at 2026-09-27 15:00:50 on dist-test-slave-pz0h"
server_key: "658a21cb8e599d7b8a74c3f39fc1efac"
server_key_iv: "b6524e8c28fb2519f6f536075f6968c3"
server_key_version: "encryptionkey@0"
I20260927 15:00:50.125849 17918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:50.143034 17918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:50.143887 17918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:50.144081 17918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:50.151438 17918 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46555
I20260927 15:00:50.151522 17984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46555 every 8 connection(s)
I20260927 15:00:50.152532 17918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:00:50.156020 17985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:00:50.159090 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 17918
I20260927 15:00:50.159241 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 15:00:50.159495 392 external_mini_cluster.cc:1428] Setting key 4fa00be1a473b751a05ee9d9b5ebc586
I20260927 15:00:50.165235 17985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27c9be341bd340c5b516fbeac9ec5052: Bootstrap starting.
I20260927 15:00:50.168274 17985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27c9be341bd340c5b516fbeac9ec5052: Neither blocks nor log segments found. Creating new log.
I20260927 15:00:50.169283 17985 log.cc:824] T 00000000000000000000000000000000 P 27c9be341bd340c5b516fbeac9ec5052: Log is configured to *not* fsync() on all Append() calls
I20260927 15:00:50.171279 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:50.161246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37834 (local address 127.0.98.62:46555)
0927 15:00:50.162051 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:50.162074 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:50.162124 (+ 50us) server_negotiation.cc:407] Connection header received
0927 15:00:50.163079 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:50.163107 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:50.163732 (+ 625us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:50.164161 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:50.164646 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:50.166108 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:50.167193 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:50.167889 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:50.168057 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:50.168915 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:50.168922 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:50.168941 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:00:50.169507 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:50.169591 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:00:50.169622 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:50.169733 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:50.169905 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 15:00:50.170218 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":138,"threads_started":1}
I20260927 15:00:50.172031 17985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27c9be341bd340c5b516fbeac9ec5052: No bootstrap required, opened a new log
I20260927 15:00:50.175998 17985 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27c9be341bd340c5b516fbeac9ec5052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27c9be341bd340c5b516fbeac9ec5052" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46555 } }
I20260927 15:00:50.176290 17985 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27c9be341bd340c5b516fbeac9ec5052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:00:50.176333 17985 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27c9be341bd340c5b516fbeac9ec5052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27c9be341bd340c5b516fbeac9ec5052, State: Initialized, Role: FOLLOWER
I20260927 15:00:50.176774 17985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27c9be341bd340c5b516fbeac9ec5052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27c9be341bd340c5b516fbeac9ec5052" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46555 } }
I20260927 15:00:50.176916 17985 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27c9be341bd340c5b516fbeac9ec5052 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:00:50.176976 17985 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27c9be341bd340c5b516fbeac9ec5052 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:00:50.177066 17985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27c9be341bd340c5b516fbeac9ec5052 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:00:50.178138 17985 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27c9be341bd340c5b516fbeac9ec5052 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27c9be341bd340c5b516fbeac9ec5052" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46555 } }
I20260927 15:00:50.178579 17985 leader_election.cc:302] T 00000000000000000000000000000000 P 27c9be341bd340c5b516fbeac9ec5052 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27c9be341bd340c5b516fbeac9ec5052; no voters:
I20260927 15:00:50.179018 17985 leader_election.cc:288] T 00000000000000000000000000000000 P 27c9be341bd340c5b516fbeac9ec5052 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:00:50.179188 17990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27c9be341bd340c5b516fbeac9ec5052 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:00:50.179366 17990 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27c9be341bd340c5b516fbeac9ec5052 [term 1 LEADER]: Becoming Leader. State: Replica: 27c9be341bd340c5b516fbeac9ec5052, State: Running, Role: LEADER
I20260927 15:00:50.179708 17990 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27c9be341bd340c5b516fbeac9ec5052 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27c9be341bd340c5b516fbeac9ec5052" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46555 } }
I20260927 15:00:50.180526 17985 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27c9be341bd340c5b516fbeac9ec5052 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:00:50.181991 17991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27c9be341bd340c5b516fbeac9ec5052 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27c9be341bd340c5b516fbeac9ec5052" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27c9be341bd340c5b516fbeac9ec5052" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46555 } } }
I20260927 15:00:50.181996 17992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27c9be341bd340c5b516fbeac9ec5052 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27c9be341bd340c5b516fbeac9ec5052. Latest consensus state: current_term: 1 leader_uuid: "27c9be341bd340c5b516fbeac9ec5052" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27c9be341bd340c5b516fbeac9ec5052" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46555 } } }
I20260927 15:00:50.182300 17991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27c9be341bd340c5b516fbeac9ec5052 [sys.catalog]: This master's current role is: LEADER
I20260927 15:00:50.182300 17992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27c9be341bd340c5b516fbeac9ec5052 [sys.catalog]: This master's current role is: LEADER
I20260927 15:00:50.183023 18003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:00:50.187124 18003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:00:50.194037 18003 catalog_manager.cc:1347] Generated new cluster ID: 9b894b945afa472d8dc156f22edc9344
I20260927 15:00:50.194161 18003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:00:50.207844 18003 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:00:50.208889 18003 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:00:50.221882 18003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27c9be341bd340c5b516fbeac9ec5052: Generated new TSK 0
I20260927 15:00:50.222666 18003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:00:50.226389 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46555
--builtin_ntp_servers=127.0.98.20:33915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:00:50.344027 18009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:50.344365 18009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:50.344415 18009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:50.348675 18009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:50.348755 18009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:50.348815 18009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:00:50.353924 18009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:50.355170 18009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:50.356606 18009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:00:50.366752 18017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:50.366746 18015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:50.367295 18014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:50.367345 18009 server_base.cc:1034] running on GCE node
I20260927 15:00:50.367830 18009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:50.368554 18009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:50.369767 18009 hybrid_clock.cc:648] HybridClock initialized: now 1790521250369736 us; error 48 us; skew 500 ppm
I20260927 15:00:50.371948 18009 webserver.cc:460] Webserver started at http://127.0.98.1:45647/ using document root <none> and password file <none>
I20260927 15:00:50.372722 18009 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:50.372814 18009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:50.373029 18009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:00:50.375336 18009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-0/data/instance:
uuid: "f3236918bc1d48f78ddeb2a7973c3c5f"
format_stamp: "Formatted at 2026-09-27 15:00:50 on dist-test-slave-pz0h"
server_key: "10dc623a676d06709207b0db6ab489db"
server_key_iv: "efb795c48d747e42016d93a88c5a15ce"
server_key_version: "encryptionkey@0"
I20260927 15:00:50.375946 18009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f3236918bc1d48f78ddeb2a7973c3c5f"
format_stamp: "Formatted at 2026-09-27 15:00:50 on dist-test-slave-pz0h"
server_key: "10dc623a676d06709207b0db6ab489db"
server_key_iv: "efb795c48d747e42016d93a88c5a15ce"
server_key_version: "encryptionkey@0"
I20260927 15:00:50.380438 18009 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:00:50.383962 18023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:50.385866 18009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:00:50.386106 18009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "f3236918bc1d48f78ddeb2a7973c3c5f"
format_stamp: "Formatted at 2026-09-27 15:00:50 on dist-test-slave-pz0h"
server_key: "10dc623a676d06709207b0db6ab489db"
server_key_iv: "efb795c48d747e42016d93a88c5a15ce"
server_key_version: "encryptionkey@0"
I20260927 15:00:50.386298 18009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:50.399231 18009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:50.400061 18009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:50.400269 18009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:50.400906 18009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:00:50.401932 18009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:00:50.402053 18009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:50.402160 18009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:00:50.402191 18009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:50.411469 18009 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38131
I20260927 15:00:50.411494 18136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38131 every 8 connection(s)
I20260927 15:00:50.412441 18009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:00:50.414359 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 18009
I20260927 15:00:50.414482 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:00:50.414765 392 external_mini_cluster.cc:1428] Setting key 3af648104d472c5ab82d9af1409ea3f1
I20260927 15:00:50.416811 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46555
--builtin_ntp_servers=127.0.98.20:33915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:00:50.422773 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:50.414340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46729 (local address 127.0.98.62:46555)
0927 15:00:50.414478 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:50.414482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:50.416003 (+ 1521us) server_negotiation.cc:407] Connection header received
0927 15:00:50.417497 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:50.417501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:50.417559 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:50.417637 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:50.418647 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:50.419294 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:50.420707 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:50.420966 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:50.421067 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:50.422264 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:50.422268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:50.422270 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:00:50.422287 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:50.422306 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:00:50.422314 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:50.422385 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:50.422528 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 15:00:50.422613 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:00:50.424211 18139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:50.414968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46555 (local address 127.0.98.1:46729)
0927 15:00:50.415818 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:50.415889 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:50.417268 (+ 1379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:50.417833 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:50.417841 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:50.417950 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:00:50.418493 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:50.418508 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:50.419438 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:50.419441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:50.420557 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:50.420576 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:50.421144 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:50.421149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:50.421331 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:50.421956 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:00:50.421958 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:00:50.422005 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:00:50.422087 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:00:50.422389 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:00:50.422399 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:00:50.422406 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 15:00:50.422521 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 15:00:50.422883 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":156,"threads_started":1}
I20260927 15:00:50.425532 18137 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46555
I20260927 15:00:50.425791 18137 heartbeater.cc:464] Registering TS with master...
I20260927 15:00:50.426368 18137 heartbeater.cc:511] Master 127.0.98.62:46555 requested a full tablet report, sending...
I20260927 15:00:50.428056 17949 ts_manager.cc:194] Registered new tserver with Master: f3236918bc1d48f78ddeb2a7973c3c5f (127.0.98.1:38131)
I20260927 15:00:50.429319 17949 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:46729
W20260927 15:00:50.533341 18140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:50.533635 18140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:50.533691 18140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:50.537856 18140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:50.537932 18140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:50.538013 18140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:00:50.543069 18140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:50.544279 18140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:50.545729 18140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:00:50.555616 18140 server_base.cc:1034] running on GCE node
W20260927 15:00:50.555543 18145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:50.555548 18148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:50.555548 18146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:50.556284 18140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:50.556989 18140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:50.558161 18140 hybrid_clock.cc:648] HybridClock initialized: now 1790521250558140 us; error 40 us; skew 500 ppm
I20260927 15:00:50.560132 18140 webserver.cc:460] Webserver started at http://127.0.98.2:37637/ using document root <none> and password file <none>
I20260927 15:00:50.560709 18140 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:50.560777 18140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:50.560969 18140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:00:50.563301 18140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-1/data/instance:
uuid: "70d7ab8b504b4e5690c9009500f89a6f"
format_stamp: "Formatted at 2026-09-27 15:00:50 on dist-test-slave-pz0h"
server_key: "ebd74a8c49c91d787e1cf3f55275b810"
server_key_iv: "02f66eca43ef45540aabf3e25c67fc74"
server_key_version: "encryptionkey@0"
I20260927 15:00:50.563875 18140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "70d7ab8b504b4e5690c9009500f89a6f"
format_stamp: "Formatted at 2026-09-27 15:00:50 on dist-test-slave-pz0h"
server_key: "ebd74a8c49c91d787e1cf3f55275b810"
server_key_iv: "02f66eca43ef45540aabf3e25c67fc74"
server_key_version: "encryptionkey@0"
I20260927 15:00:50.567799 18140 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:00:50.570374 18154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:50.571712 18140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:00:50.571838 18140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "70d7ab8b504b4e5690c9009500f89a6f"
format_stamp: "Formatted at 2026-09-27 15:00:50 on dist-test-slave-pz0h"
server_key: "ebd74a8c49c91d787e1cf3f55275b810"
server_key_iv: "02f66eca43ef45540aabf3e25c67fc74"
server_key_version: "encryptionkey@0"
I20260927 15:00:50.571956 18140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:50.597180 18140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:50.597944 18140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:50.598134 18140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:50.598770 18140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:00:50.599831 18140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:00:50.599891 18140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:50.599948 18140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:00:50.599974 18140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:50.608428 18140 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46349
I20260927 15:00:50.608449 18267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46349 every 8 connection(s)
I20260927 15:00:50.609418 18140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:00:50.613651 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 18140
I20260927 15:00:50.613776 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:00:50.614069 392 external_mini_cluster.cc:1428] Setting key c1fd60a663e337525436d9df785f923a
I20260927 15:00:50.616552 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46555
--builtin_ntp_servers=127.0.98.20:33915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:00:50.620631 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:50.611227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43125 (local address 127.0.98.62:46555)
0927 15:00:50.611361 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:50.611364 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:50.612438 (+ 1074us) server_negotiation.cc:407] Connection header received
0927 15:00:50.614010 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:50.614014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:50.614076 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:50.614187 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:50.615445 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:50.616117 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:50.618039 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:50.618393 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:50.618525 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:50.619991 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:50.619996 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:50.619998 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:00:50.620017 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:50.620041 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:00:50.620052 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:50.620137 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:50.620370 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 15:00:50.620456 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:00:50.621567 18270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:50.611620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46555 (local address 127.0.98.2:43125)
0927 15:00:50.612279 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:50.612320 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:50.613734 (+ 1414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:50.614439 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:50.614450 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:50.614567 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:00:50.615251 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:50.615274 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:50.616275 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:50.616280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:50.617854 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:50.617871 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:50.618653 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:50.618665 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:50.618891 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:50.619662 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:00:50.619663 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:00:50.619713 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:00:50.619798 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:00:50.620176 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:00:50.620187 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:00:50.620196 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 15:00:50.620352 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 15:00:50.620647 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":116,"threads_started":1}
I20260927 15:00:50.623772 18268 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46555
I20260927 15:00:50.624166 18268 heartbeater.cc:464] Registering TS with master...
I20260927 15:00:50.624794 18268 heartbeater.cc:511] Master 127.0.98.62:46555 requested a full tablet report, sending...
I20260927 15:00:50.626070 17949 ts_manager.cc:194] Registered new tserver with Master: 70d7ab8b504b4e5690c9009500f89a6f (127.0.98.2:46349)
I20260927 15:00:50.626695 17949 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:43125
W20260927 15:00:50.732589 18271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:50.732971 18271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:50.733029 18271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:50.737318 18271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:50.737408 18271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:50.737468 18271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:00:50.742607 18271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:50.743803 18271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:50.745220 18271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:00:50.755632 18277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:50.755620 18279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:50.755618 18276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:50.755755 18271 server_base.cc:1034] running on GCE node
I20260927 15:00:50.756332 18271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:50.757018 18271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:50.758206 18271 hybrid_clock.cc:648] HybridClock initialized: now 1790521250758182 us; error 40 us; skew 500 ppm
I20260927 15:00:50.760202 18271 webserver.cc:460] Webserver started at http://127.0.98.3:43487/ using document root <none> and password file <none>
I20260927 15:00:50.760807 18271 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:50.760877 18271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:50.761067 18271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:00:50.763243 18271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-2/data/instance:
uuid: "aa7b4e315fba40ce9ab8096e61c5ff48"
format_stamp: "Formatted at 2026-09-27 15:00:50 on dist-test-slave-pz0h"
server_key: "2397958bda180170387fecf65274492a"
server_key_iv: "70e69212c5e768692be9c6b4bd283792"
server_key_version: "encryptionkey@0"
I20260927 15:00:50.763830 18271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "aa7b4e315fba40ce9ab8096e61c5ff48"
format_stamp: "Formatted at 2026-09-27 15:00:50 on dist-test-slave-pz0h"
server_key: "2397958bda180170387fecf65274492a"
server_key_iv: "70e69212c5e768692be9c6b4bd283792"
server_key_version: "encryptionkey@0"
I20260927 15:00:50.767895 18271 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:00:50.770454 18285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:50.771718 18271 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:00:50.771853 18271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "aa7b4e315fba40ce9ab8096e61c5ff48"
format_stamp: "Formatted at 2026-09-27 15:00:50 on dist-test-slave-pz0h"
server_key: "2397958bda180170387fecf65274492a"
server_key_iv: "70e69212c5e768692be9c6b4bd283792"
server_key_version: "encryptionkey@0"
I20260927 15:00:50.771965 18271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:50.792582 18271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:50.793324 18271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:50.793507 18271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:50.794114 18271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:00:50.795106 18271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:00:50.795156 18271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:50.795212 18271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:00:50.795236 18271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:50.803436 18271 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39333
I20260927 15:00:50.803484 18398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39333 every 8 connection(s)
I20260927 15:00:50.804339 18271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:00:50.813388 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:50.806187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35843 (local address 127.0.98.62:46555)
0927 15:00:50.806361 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:50.806365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:50.807444 (+ 1079us) server_negotiation.cc:407] Connection header received
0927 15:00:50.808879 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:50.808882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:50.808932 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:50.809011 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:50.809941 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:50.810474 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:50.811639 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:50.811871 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:50.811953 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:50.812996 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:50.812999 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:50.813001 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:00:50.813014 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:50.813031 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:00:50.813040 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:50.813094 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:50.813202 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 15:00:50.813259 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 15:00:50.813566 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 18271
I20260927 15:00:50.813668 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-2/wal/instance
I20260927 15:00:50.813968 392 external_mini_cluster.cc:1428] Setting key 09bdbfa1f0322b5a1255c6dc785e6300
I20260927 15:00:50.814527 18401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:50.806579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46555 (local address 127.0.98.3:35843)
0927 15:00:50.807305 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:50.807349 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:50.808650 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:50.809179 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:50.809186 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:50.809293 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:00:50.809802 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:50.809815 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:50.810581 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:50.810583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:50.811522 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:50.811534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:50.811998 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:50.812001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:50.812142 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:50.812738 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:00:50.812740 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:00:50.812787 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:00:50.812861 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:00:50.813100 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:00:50.813105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:00:50.813109 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 15:00:50.813197 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 15:00:50.813534 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":156,"threads_started":1}
I20260927 15:00:50.815554 18399 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46555
I20260927 15:00:50.815796 18399 heartbeater.cc:464] Registering TS with master...
I20260927 15:00:50.816323 18399 heartbeater.cc:511] Master 127.0.98.62:46555 requested a full tablet report, sending...
I20260927 15:00:50.817325 17949 ts_manager.cc:194] Registered new tserver with Master: aa7b4e315fba40ce9ab8096e61c5ff48 (127.0.98.3:39333)
I20260927 15:00:50.817866 17949 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:35843
I20260927 15:00:50.828047 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:00:50.833994 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:50.830074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37840 (local address 127.0.98.62:46555)
0927 15:00:50.830202 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:50.830205 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:50.830496 (+ 291us) server_negotiation.cc:407] Connection header received
0927 15:00:50.830558 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:50.830560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:50.830612 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:50.830708 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:50.831110 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:50.831659 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:50.832516 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:50.832733 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:50.832814 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:50.833607 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:50.833609 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:50.833611 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:00:50.833623 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:50.833637 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:00:50.833644 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:50.833690 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:50.833825 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 15:00:50.833876 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:00:50.836923 17949 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:00:50.839972 17949 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:00:50.854668 18410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:50.848584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46349 (local address 127.0.0.1:42146)
0927 15:00:50.849226 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:50.849276 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:50.849418 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:50.850222 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:50.850227 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:50.850261 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:50.850635 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:50.850642 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:50.852004 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:50.852007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:50.853180 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:50.853189 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:50.854199 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:50.854203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:50.854308 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:50.854360 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 15:00:50.854445 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 15:00:50.854538 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":292,"spinlock_wait_cycles":2176,"thread_start_us":114,"threads_started":1}
I20260927 15:00:50.854770 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:50.848721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42146 (local address 127.0.98.2:46349)
0927 15:00:50.849720 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:50.849731 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:50.849755 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:00:50.849829 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:50.849834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:50.850050 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:50.850376 (+ 326us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:50.850756 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:50.851911 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:50.853305 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:50.854079 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:50.854193 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:50.854405 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:50.854497 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 15:00:50.854601 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":894,"thread_start_us":134,"threads_started":1}
I20260927 15:00:50.856676 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:50.849752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34952 (local address 127.0.98.1:38131)
0927 15:00:50.850240 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:50.850250 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:50.850272 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:00:50.850340 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:50.850344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:50.850504 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:50.850634 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:50.850876 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:50.851586 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:50.854288 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:50.855785 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:50.855891 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:50.856079 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:50.856398 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 15:00:50.856509 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":122,"threads_started":1}
I20260927 15:00:50.856710 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:50.848667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39333 (local address 127.0.0.1:58338)
0927 15:00:50.849326 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:50.849340 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:50.849445 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:50.850219 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:50.850225 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:50.850260 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:50.850455 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:50.850462 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:50.851919 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:50.851923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:50.853373 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:50.853384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:50.856251 (+ 2867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:50.856255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:50.856399 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:50.856423 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:00:50.856500 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 15:00:50.856572 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":136,"spinlock_wait_cycles":7168,"thread_start_us":108,"threads_started":1}
I20260927 15:00:50.856760 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:50.849020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58338 (local address 127.0.98.3:39333)
0927 15:00:50.849719 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:50.849731 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:50.849755 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:00:50.849831 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:50.849836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:50.850050 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:50.850359 (+ 309us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:50.850568 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:50.851806 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:50.853498 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:50.856121 (+ 2623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:50.856223 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:50.856430 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:50.856524 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 15:00:50.856618 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":104,"threads_started":1}
I20260927 15:00:50.856980 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:50.849342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38131 (local address 127.0.0.1:34952)
0927 15:00:50.850121 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:50.850134 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:50.850206 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:50.850635 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:50.850638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:50.850654 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:50.850745 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:50.850752 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:50.851698 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:50.851701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:50.853378 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:50.853384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:50.855900 (+ 2516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:50.855904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:50.856103 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:50.856192 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 15:00:50.856252 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 15:00:50.856865 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"mutex_wait_us":932,"thread_start_us":94,"threads_started":1}
I20260927 15:00:50.857928 18202 tablet_service.cc:1467] Processing CreateTablet for tablet 38eb9eafb6fb44f9a41003f5bd7ab929 (DEFAULT_TABLE table=test-table [id=c64c436907e9407b8f90b455330be882]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:00:50.858947 18202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38eb9eafb6fb44f9a41003f5bd7ab929. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:00:50.859681 18071 tablet_service.cc:1467] Processing CreateTablet for tablet 38eb9eafb6fb44f9a41003f5bd7ab929 (DEFAULT_TABLE table=test-table [id=c64c436907e9407b8f90b455330be882]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:00:50.859681 18333 tablet_service.cc:1467] Processing CreateTablet for tablet 38eb9eafb6fb44f9a41003f5bd7ab929 (DEFAULT_TABLE table=test-table [id=c64c436907e9407b8f90b455330be882]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:00:50.860817 18071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38eb9eafb6fb44f9a41003f5bd7ab929. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:00:50.860817 18333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38eb9eafb6fb44f9a41003f5bd7ab929. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:00:50.866391 18416 tablet_bootstrap.cc:492] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f: Bootstrap starting.
I20260927 15:00:50.868374 18418 tablet_bootstrap.cc:492] T 38eb9eafb6fb44f9a41003f5bd7ab929 P f3236918bc1d48f78ddeb2a7973c3c5f: Bootstrap starting.
I20260927 15:00:50.868942 18416 tablet_bootstrap.cc:654] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f: Neither blocks nor log segments found. Creating new log.
I20260927 15:00:50.869944 18416 log.cc:824] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:00:50.870673 18418 tablet_bootstrap.cc:654] T 38eb9eafb6fb44f9a41003f5bd7ab929 P f3236918bc1d48f78ddeb2a7973c3c5f: Neither blocks nor log segments found. Creating new log.
I20260927 15:00:50.870766 18417 tablet_bootstrap.cc:492] T 38eb9eafb6fb44f9a41003f5bd7ab929 P aa7b4e315fba40ce9ab8096e61c5ff48: Bootstrap starting.
I20260927 15:00:50.871580 18418 log.cc:824] T 38eb9eafb6fb44f9a41003f5bd7ab929 P f3236918bc1d48f78ddeb2a7973c3c5f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:00:50.872416 18416 tablet_bootstrap.cc:492] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f: No bootstrap required, opened a new log
I20260927 15:00:50.872632 18416 ts_tablet_manager.cc:1397] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:00:50.873005 18417 tablet_bootstrap.cc:654] T 38eb9eafb6fb44f9a41003f5bd7ab929 P aa7b4e315fba40ce9ab8096e61c5ff48: Neither blocks nor log segments found. Creating new log.
I20260927 15:00:50.873797 18418 tablet_bootstrap.cc:492] T 38eb9eafb6fb44f9a41003f5bd7ab929 P f3236918bc1d48f78ddeb2a7973c3c5f: No bootstrap required, opened a new log
I20260927 15:00:50.873848 18417 log.cc:824] T 38eb9eafb6fb44f9a41003f5bd7ab929 P aa7b4e315fba40ce9ab8096e61c5ff48: Log is configured to *not* fsync() on all Append() calls
I20260927 15:00:50.874006 18418 ts_tablet_manager.cc:1397] T 38eb9eafb6fb44f9a41003f5bd7ab929 P f3236918bc1d48f78ddeb2a7973c3c5f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:00:50.876008 18417 tablet_bootstrap.cc:492] T 38eb9eafb6fb44f9a41003f5bd7ab929 P aa7b4e315fba40ce9ab8096e61c5ff48: No bootstrap required, opened a new log
I20260927 15:00:50.876209 18417 ts_tablet_manager.cc:1397] T 38eb9eafb6fb44f9a41003f5bd7ab929 P aa7b4e315fba40ce9ab8096e61c5ff48: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 15:00:50.876499 18416 raft_consensus.cc:348] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d7ab8b504b4e5690c9009500f89a6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46349 } } peers { permanent_uuid: "aa7b4e315fba40ce9ab8096e61c5ff48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39333 } } peers { permanent_uuid: "f3236918bc1d48f78ddeb2a7973c3c5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38131 } }
I20260927 15:00:50.876763 18416 raft_consensus.cc:374] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:00:50.876816 18416 raft_consensus.cc:729] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70d7ab8b504b4e5690c9009500f89a6f, State: Initialized, Role: FOLLOWER
I20260927 15:00:50.877336 18416 consensus_queue.cc:260] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d7ab8b504b4e5690c9009500f89a6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46349 } } peers { permanent_uuid: "aa7b4e315fba40ce9ab8096e61c5ff48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39333 } } peers { permanent_uuid: "f3236918bc1d48f78ddeb2a7973c3c5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38131 } }
I20260927 15:00:50.878438 18416 ts_tablet_manager.cc:1428] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:00:50.878278 18418 raft_consensus.cc:348] T 38eb9eafb6fb44f9a41003f5bd7ab929 P f3236918bc1d48f78ddeb2a7973c3c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d7ab8b504b4e5690c9009500f89a6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46349 } } peers { permanent_uuid: "aa7b4e315fba40ce9ab8096e61c5ff48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39333 } } peers { permanent_uuid: "f3236918bc1d48f78ddeb2a7973c3c5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38131 } }
I20260927 15:00:50.878535 18418 raft_consensus.cc:374] T 38eb9eafb6fb44f9a41003f5bd7ab929 P f3236918bc1d48f78ddeb2a7973c3c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:00:50.878594 18418 raft_consensus.cc:729] T 38eb9eafb6fb44f9a41003f5bd7ab929 P f3236918bc1d48f78ddeb2a7973c3c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3236918bc1d48f78ddeb2a7973c3c5f, State: Initialized, Role: FOLLOWER
I20260927 15:00:50.878865 18268 heartbeater.cc:503] Master 127.0.98.62:46555 was elected leader, sending a full tablet report...
I20260927 15:00:50.879117 18418 consensus_queue.cc:260] T 38eb9eafb6fb44f9a41003f5bd7ab929 P f3236918bc1d48f78ddeb2a7973c3c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d7ab8b504b4e5690c9009500f89a6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46349 } } peers { permanent_uuid: "aa7b4e315fba40ce9ab8096e61c5ff48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39333 } } peers { permanent_uuid: "f3236918bc1d48f78ddeb2a7973c3c5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38131 } }
I20260927 15:00:50.879827 18137 heartbeater.cc:503] Master 127.0.98.62:46555 was elected leader, sending a full tablet report...
I20260927 15:00:50.880152 18418 ts_tablet_manager.cc:1428] T 38eb9eafb6fb44f9a41003f5bd7ab929 P f3236918bc1d48f78ddeb2a7973c3c5f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 15:00:50.880396 18417 raft_consensus.cc:348] T 38eb9eafb6fb44f9a41003f5bd7ab929 P aa7b4e315fba40ce9ab8096e61c5ff48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d7ab8b504b4e5690c9009500f89a6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46349 } } peers { permanent_uuid: "aa7b4e315fba40ce9ab8096e61c5ff48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39333 } } peers { permanent_uuid: "f3236918bc1d48f78ddeb2a7973c3c5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38131 } }
I20260927 15:00:50.880609 18417 raft_consensus.cc:374] T 38eb9eafb6fb44f9a41003f5bd7ab929 P aa7b4e315fba40ce9ab8096e61c5ff48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:00:50.880648 18417 raft_consensus.cc:729] T 38eb9eafb6fb44f9a41003f5bd7ab929 P aa7b4e315fba40ce9ab8096e61c5ff48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa7b4e315fba40ce9ab8096e61c5ff48, State: Initialized, Role: FOLLOWER
I20260927 15:00:50.881048 18417 consensus_queue.cc:260] T 38eb9eafb6fb44f9a41003f5bd7ab929 P aa7b4e315fba40ce9ab8096e61c5ff48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d7ab8b504b4e5690c9009500f89a6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46349 } } peers { permanent_uuid: "aa7b4e315fba40ce9ab8096e61c5ff48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39333 } } peers { permanent_uuid: "f3236918bc1d48f78ddeb2a7973c3c5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38131 } }
I20260927 15:00:50.881846 18399 heartbeater.cc:503] Master 127.0.98.62:46555 was elected leader, sending a full tablet report...
I20260927 15:00:50.882033 18417 ts_tablet_manager.cc:1428] T 38eb9eafb6fb44f9a41003f5bd7ab929 P aa7b4e315fba40ce9ab8096e61c5ff48: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260927 15:00:50.913719 18138 tablet.cc:2369] T 38eb9eafb6fb44f9a41003f5bd7ab929 P f3236918bc1d48f78ddeb2a7973c3c5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:00:51.055356 18400 tablet.cc:2369] T 38eb9eafb6fb44f9a41003f5bd7ab929 P aa7b4e315fba40ce9ab8096e61c5ff48: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:00:51.110602 18269 tablet.cc:2369] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:00:51.144088 18422 raft_consensus.cc:482] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:00:51.144331 18422 raft_consensus.cc:504] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d7ab8b504b4e5690c9009500f89a6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46349 } } peers { permanent_uuid: "aa7b4e315fba40ce9ab8096e61c5ff48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39333 } } peers { permanent_uuid: "f3236918bc1d48f78ddeb2a7973c3c5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38131 } }
I20260927 15:00:51.145504 18422 leader_election.cc:288] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa7b4e315fba40ce9ab8096e61c5ff48 (127.0.98.3:39333), f3236918bc1d48f78ddeb2a7973c3c5f (127.0.98.1:38131)
I20260927 15:00:51.149454 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:51.145901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33873 (local address 127.0.98.1:38131)
0927 15:00:51.146053 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:51.146058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:51.146218 (+ 160us) server_negotiation.cc:407] Connection header received
0927 15:00:51.146384 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:51.146387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:51.146431 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:51.146493 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:51.146760 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:51.147299 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:51.148541 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:51.148933 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:51.149024 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:51.149087 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:51.149260 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 15:00:51.149331 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:00:51.149499 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:51.145757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38131 (local address 127.0.98.2:33873)
0927 15:00:51.146134 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:51.146150 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:51.146261 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:51.146523 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:51.146526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:51.146541 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:51.146659 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:51.146664 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:51.147409 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:51.147413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:51.148410 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:51.148421 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:51.149046 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:51.149050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:51.149148 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:51.149171 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:00:51.149231 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 15:00:51.149304 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":86,"thread_start_us":95,"threads_started":1}
I20260927 15:00:51.149528 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:51.146047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34771 (local address 127.0.98.3:39333)
0927 15:00:51.146173 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:51.146177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:51.146311 (+ 134us) server_negotiation.cc:407] Connection header received
0927 15:00:51.146441 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:51.146444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:51.146484 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:51.146546 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:51.146827 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:51.147317 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:51.148566 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:51.148983 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:51.149076 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:51.149138 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:51.149345 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 15:00:51.149424 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 15:00:51.149545 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:51.145757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39333 (local address 127.0.98.2:34771)
0927 15:00:51.146213 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:51.146227 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:51.146341 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:51.146572 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:51.146574 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:51.146589 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:51.146700 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:51.146705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:51.147426 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:51.147430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:51.148438 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:51.148449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:51.149108 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:51.149112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:51.149234 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:51.149251 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 15:00:51.149315 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 15:00:51.149382 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":105,"thread_start_us":86,"threads_started":1}
I20260927 15:00:51.150084 18091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38eb9eafb6fb44f9a41003f5bd7ab929" candidate_uuid: "70d7ab8b504b4e5690c9009500f89a6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3236918bc1d48f78ddeb2a7973c3c5f" is_pre_election: true
I20260927 15:00:51.150094 18353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38eb9eafb6fb44f9a41003f5bd7ab929" candidate_uuid: "70d7ab8b504b4e5690c9009500f89a6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa7b4e315fba40ce9ab8096e61c5ff48" is_pre_election: true
I20260927 15:00:51.150364 18091 raft_consensus.cc:2454] T 38eb9eafb6fb44f9a41003f5bd7ab929 P f3236918bc1d48f78ddeb2a7973c3c5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70d7ab8b504b4e5690c9009500f89a6f in term 0.
I20260927 15:00:51.150364 18353 raft_consensus.cc:2454] T 38eb9eafb6fb44f9a41003f5bd7ab929 P aa7b4e315fba40ce9ab8096e61c5ff48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70d7ab8b504b4e5690c9009500f89a6f in term 0.
I20260927 15:00:51.150861 18156 leader_election.cc:302] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70d7ab8b504b4e5690c9009500f89a6f, aa7b4e315fba40ce9ab8096e61c5ff48; no voters:
I20260927 15:00:51.151110 18422 raft_consensus.cc:2781] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:00:51.151196 18422 raft_consensus.cc:482] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:00:51.151238 18422 raft_consensus.cc:3037] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:00:51.152264 18422 raft_consensus.cc:504] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d7ab8b504b4e5690c9009500f89a6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46349 } } peers { permanent_uuid: "aa7b4e315fba40ce9ab8096e61c5ff48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39333 } } peers { permanent_uuid: "f3236918bc1d48f78ddeb2a7973c3c5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38131 } }
I20260927 15:00:51.152637 18422 leader_election.cc:288] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f [CANDIDATE]: Term 1 election: Requested vote from peers aa7b4e315fba40ce9ab8096e61c5ff48 (127.0.98.3:39333), f3236918bc1d48f78ddeb2a7973c3c5f (127.0.98.1:38131)
I20260927 15:00:51.153031 18353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38eb9eafb6fb44f9a41003f5bd7ab929" candidate_uuid: "70d7ab8b504b4e5690c9009500f89a6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa7b4e315fba40ce9ab8096e61c5ff48"
I20260927 15:00:51.153172 18353 raft_consensus.cc:3037] T 38eb9eafb6fb44f9a41003f5bd7ab929 P aa7b4e315fba40ce9ab8096e61c5ff48 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:00:51.154464 18353 raft_consensus.cc:2454] T 38eb9eafb6fb44f9a41003f5bd7ab929 P aa7b4e315fba40ce9ab8096e61c5ff48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70d7ab8b504b4e5690c9009500f89a6f in term 1.
I20260927 15:00:51.154755 18091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38eb9eafb6fb44f9a41003f5bd7ab929" candidate_uuid: "70d7ab8b504b4e5690c9009500f89a6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3236918bc1d48f78ddeb2a7973c3c5f"
I20260927 15:00:51.154835 18156 leader_election.cc:302] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70d7ab8b504b4e5690c9009500f89a6f, aa7b4e315fba40ce9ab8096e61c5ff48; no voters:
I20260927 15:00:51.154925 18091 raft_consensus.cc:3037] T 38eb9eafb6fb44f9a41003f5bd7ab929 P f3236918bc1d48f78ddeb2a7973c3c5f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:00:51.155015 18422 raft_consensus.cc:2781] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:00:51.155238 18422 raft_consensus.cc:686] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f [term 1 LEADER]: Becoming Leader. State: Replica: 70d7ab8b504b4e5690c9009500f89a6f, State: Running, Role: LEADER
I20260927 15:00:51.155614 18422 consensus_queue.cc:237] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d7ab8b504b4e5690c9009500f89a6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46349 } } peers { permanent_uuid: "aa7b4e315fba40ce9ab8096e61c5ff48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39333 } } peers { permanent_uuid: "f3236918bc1d48f78ddeb2a7973c3c5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38131 } }
I20260927 15:00:51.156180 18091 raft_consensus.cc:2454] T 38eb9eafb6fb44f9a41003f5bd7ab929 P f3236918bc1d48f78ddeb2a7973c3c5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70d7ab8b504b4e5690c9009500f89a6f in term 1.
I20260927 15:00:51.158718 17947 catalog_manager.cc:5495] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 70d7ab8b504b4e5690c9009500f89a6f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "70d7ab8b504b4e5690c9009500f89a6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d7ab8b504b4e5690c9009500f89a6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa7b4e315fba40ce9ab8096e61c5ff48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3236918bc1d48f78ddeb2a7973c3c5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38131 } health_report { overall_health: UNKNOWN } } }
I20260927 15:00:51.218499 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:51.215014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42158 (local address 127.0.98.2:46349)
0927 15:00:51.215151 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:51.215156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:51.215171 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:00:51.215285 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:51.215288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:51.215346 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:51.215442 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:00:51.215690 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:51.216201 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:51.217028 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:51.217244 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:51.217335 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:51.217646 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:00:51.218101 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:00:51.218213 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:51.218286 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 15:00:51.218348 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:00:51.225936 18091 raft_consensus.cc:1261] T 38eb9eafb6fb44f9a41003f5bd7ab929 P f3236918bc1d48f78ddeb2a7973c3c5f [term 1 FOLLOWER]: Refusing update from remote peer 70d7ab8b504b4e5690c9009500f89a6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:00:51.225936 18353 raft_consensus.cc:1261] T 38eb9eafb6fb44f9a41003f5bd7ab929 P aa7b4e315fba40ce9ab8096e61c5ff48 [term 1 FOLLOWER]: Refusing update from remote peer 70d7ab8b504b4e5690c9009500f89a6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:00:51.226969 18422 consensus_queue.cc:1035] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3236918bc1d48f78ddeb2a7973c3c5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:00:51.227208 18427 consensus_queue.cc:1035] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7b4e315fba40ce9ab8096e61c5ff48" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:00:51.236867 18431 mvcc.cc:205] Tried to move back new op lower bound from 7333975045009522688 to 7333975044735979520. Current Snapshot: MvccSnapshot[applied={T|T < 7333975045009522688}]
I20260927 15:00:51.247198 17948 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:00:51.247421 17948 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:00:51.250156 17948 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 27c9be341bd340c5b516fbeac9ec5052: Sending DeleteTablet for 3 replicas of tablet 38eb9eafb6fb44f9a41003f5bd7ab929
I20260927 15:00:51.251137 18071 tablet_service.cc:1514] Processing DeleteTablet for tablet 38eb9eafb6fb44f9a41003f5bd7ab929 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:00:51 UTC) from {username='slave'} at 127.0.0.1:34952
I20260927 15:00:51.251129 18202 tablet_service.cc:1514] Processing DeleteTablet for tablet 38eb9eafb6fb44f9a41003f5bd7ab929 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:00:51 UTC) from {username='slave'} at 127.0.0.1:42146
I20260927 15:00:51.251137 18333 tablet_service.cc:1514] Processing DeleteTablet for tablet 38eb9eafb6fb44f9a41003f5bd7ab929 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:00:51 UTC) from {username='slave'} at 127.0.0.1:58338
I20260927 15:00:51.251710 18441 tablet_replica.cc:330] T 38eb9eafb6fb44f9a41003f5bd7ab929 P f3236918bc1d48f78ddeb2a7973c3c5f: stopping tablet replica
I20260927 15:00:51.251883 18442 tablet_replica.cc:330] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f: stopping tablet replica
I20260927 15:00:51.252090 18441 raft_consensus.cc:2229] T 38eb9eafb6fb44f9a41003f5bd7ab929 P f3236918bc1d48f78ddeb2a7973c3c5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:00:51.252195 18440 tablet_replica.cc:330] T 38eb9eafb6fb44f9a41003f5bd7ab929 P aa7b4e315fba40ce9ab8096e61c5ff48: stopping tablet replica
I20260927 15:00:51.252465 18441 raft_consensus.cc:2258] T 38eb9eafb6fb44f9a41003f5bd7ab929 P f3236918bc1d48f78ddeb2a7973c3c5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:00:51.252507 18440 raft_consensus.cc:2229] T 38eb9eafb6fb44f9a41003f5bd7ab929 P aa7b4e315fba40ce9ab8096e61c5ff48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:00:51.252717 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 18009
I20260927 15:00:51.252801 18440 raft_consensus.cc:2258] T 38eb9eafb6fb44f9a41003f5bd7ab929 P aa7b4e315fba40ce9ab8096e61c5ff48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:00:51.255175 18440 ts_tablet_manager.cc:1905] T 38eb9eafb6fb44f9a41003f5bd7ab929 P aa7b4e315fba40ce9ab8096e61c5ff48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:00:51.255282 18442 raft_consensus.cc:2229] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:00:51.255720 18442 raft_consensus.cc:2258] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:00:51.258066 18440 ts_tablet_manager.cc:1918] T 38eb9eafb6fb44f9a41003f5bd7ab929 P aa7b4e315fba40ce9ab8096e61c5ff48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:00:51.258175 18440 log.cc:1196] T 38eb9eafb6fb44f9a41003f5bd7ab929 P aa7b4e315fba40ce9ab8096e61c5ff48: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790521004237705-392-0/minicluster-data/ts-2/wal/wals/38eb9eafb6fb44f9a41003f5bd7ab929
I20260927 15:00:51.258459 18440 ts_tablet_manager.cc:1939] T 38eb9eafb6fb44f9a41003f5bd7ab929 P aa7b4e315fba40ce9ab8096e61c5ff48: Deleting consensus metadata
I20260927 15:00:51.258874 18442 ts_tablet_manager.cc:1905] T 38eb9eafb6fb44f9a41003f5bd7ab929 P 70d7ab8b504b4e5690c9009500f89a6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:00:51.259853 17934 catalog_manager.cc:4841] TS aa7b4e315fba40ce9ab8096e61c5ff48 (127.0.98.3:39333): tablet 38eb9eafb6fb44f9a41003f5bd7ab929 (table test-table [id=c64c436907e9407b8f90b455330be882]) successfully deleted
W20260927 15:00:51.260206 17936 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38131 (error 108)
I20260927 15:00:51.260447 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 18140
I20260927 15:00:51.262101 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:51.261769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38131 (local address 127.0.0.1:34954)
0927 15:00:51.261946 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:51.262013 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260927 15:00:51.262279 17936 catalog_manager.cc:4584] TS f3236918bc1d48f78ddeb2a7973c3c5f (127.0.98.1:38131): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 38eb9eafb6fb44f9a41003f5bd7ab929: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38131: connect: Connection refused (error 111)
I20260927 15:00:51.266510 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 18271
I20260927 15:00:51.267284 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:51.266678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46349 (local address 127.0.0.1:42174)
0927 15:00:51.267125 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:51.267185 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359}
I20260927 15:00:51.271979 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 17918
2026-09-27T15:00:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1350 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:00:51 dist-test-slave-pz0h krb5kdc[18446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:00:51 dist-test-slave-pz0h krb5kdc[18446](info): set up 2 sockets
Sep 27 15:00:51 dist-test-slave-pz0h krb5kdc[18446](info): commencing operation
krb5kdc: starting...
W20260927 15:00:52.234524 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 15:00:52 dist-test-slave-pz0h krb5kdc[18446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:00:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:00:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:00:52.354117 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44219
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:00:52.467257 18459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:52.467555 18459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:52.467617 18459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:52.471925 18459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:00:52.472009 18459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:52.472035 18459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:52.472057 18459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:00:52.472085 18459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:00:52.477823 18459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:52.479005 18459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:52.480070 18459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:00:52.488695 18464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:52.488741 18459 server_base.cc:1034] running on GCE node
W20260927 15:00:52.488683 18465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:52.488678 18467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:52.489563 18459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:52.490685 18459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:52.491874 18459 hybrid_clock.cc:648] HybridClock initialized: now 1790521252491856 us; error 38 us; skew 500 ppm
Sep 27 15:00:52 dist-test-slave-pz0h krb5kdc[18446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:00:52.498471 18459 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:00:52.499819 18459 webserver.cc:460] Webserver started at http://127.0.98.62:40893/ using document root <none> and password file <none>
I20260927 15:00:52.500602 18459 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:52.500698 18459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:52.500945 18459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:00:52.503201 18459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "9631cb1f94884976a034064808f1b9af"
format_stamp: "Formatted at 2026-09-27 15:00:52 on dist-test-slave-pz0h"
server_key: "7b6fd43b9096a1221a0ddf1f70582767"
server_key_iv: "e5a68b976af8530bdc05db02fd5be596"
server_key_version: "encryptionkey@0"
I20260927 15:00:52.503800 18459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "9631cb1f94884976a034064808f1b9af"
format_stamp: "Formatted at 2026-09-27 15:00:52 on dist-test-slave-pz0h"
server_key: "7b6fd43b9096a1221a0ddf1f70582767"
server_key_iv: "e5a68b976af8530bdc05db02fd5be596"
server_key_version: "encryptionkey@0"
I20260927 15:00:52.507665 18459 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:00:52.510159 18474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:52.511554 18459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:00:52.511695 18459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "9631cb1f94884976a034064808f1b9af"
format_stamp: "Formatted at 2026-09-27 15:00:52 on dist-test-slave-pz0h"
server_key: "7b6fd43b9096a1221a0ddf1f70582767"
server_key_iv: "e5a68b976af8530bdc05db02fd5be596"
server_key_version: "encryptionkey@0"
I20260927 15:00:52.511814 18459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:52.550940 18459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:52.554102 18459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:52.554302 18459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:52.560976 18459 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44219
I20260927 15:00:52.560989 18526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44219 every 8 connection(s)
I20260927 15:00:52.561937 18459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:00:52.564719 18527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:00:52.570436 18527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9631cb1f94884976a034064808f1b9af: Bootstrap starting.
I20260927 15:00:52.571573 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 18459
I20260927 15:00:52.571715 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 15:00:52.572007 392 external_mini_cluster.cc:1428] Setting key 5145fe11babc8b083027f5355a720d4d
I20260927 15:00:52.573472 18527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9631cb1f94884976a034064808f1b9af: Neither blocks nor log segments found. Creating new log.
I20260927 15:00:52.574203 18527 log.cc:824] T 00000000000000000000000000000000 P 9631cb1f94884976a034064808f1b9af: Log is configured to *not* fsync() on all Append() calls
I20260927 15:00:52.576436 18527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9631cb1f94884976a034064808f1b9af: No bootstrap required, opened a new log
I20260927 15:00:52.580827 18527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9631cb1f94884976a034064808f1b9af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9631cb1f94884976a034064808f1b9af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44219 } }
Sep 27 15:00:52 dist-test-slave-pz0h krb5kdc[18446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:00:52.581086 18527 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9631cb1f94884976a034064808f1b9af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:00:52.581153 18527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9631cb1f94884976a034064808f1b9af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9631cb1f94884976a034064808f1b9af, State: Initialized, Role: FOLLOWER
I20260927 15:00:52.581707 18527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9631cb1f94884976a034064808f1b9af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9631cb1f94884976a034064808f1b9af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44219 } }
I20260927 15:00:52.581867 18527 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9631cb1f94884976a034064808f1b9af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:00:52.581938 18527 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9631cb1f94884976a034064808f1b9af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:00:52.582038 18527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9631cb1f94884976a034064808f1b9af [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:00:52.583033 18527 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9631cb1f94884976a034064808f1b9af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9631cb1f94884976a034064808f1b9af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44219 } }
I20260927 15:00:52.583386 18527 leader_election.cc:302] T 00000000000000000000000000000000 P 9631cb1f94884976a034064808f1b9af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9631cb1f94884976a034064808f1b9af; no voters:
I20260927 15:00:52.583689 18527 leader_election.cc:288] T 00000000000000000000000000000000 P 9631cb1f94884976a034064808f1b9af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:00:52.583820 18532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9631cb1f94884976a034064808f1b9af [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:00:52.584022 18532 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9631cb1f94884976a034064808f1b9af [term 1 LEADER]: Becoming Leader. State: Replica: 9631cb1f94884976a034064808f1b9af, State: Running, Role: LEADER
I20260927 15:00:52.584383 18532 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9631cb1f94884976a034064808f1b9af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9631cb1f94884976a034064808f1b9af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44219 } }
I20260927 15:00:52.584978 18527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9631cb1f94884976a034064808f1b9af [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:00:52.586215 18534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9631cb1f94884976a034064808f1b9af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9631cb1f94884976a034064808f1b9af. Latest consensus state: current_term: 1 leader_uuid: "9631cb1f94884976a034064808f1b9af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9631cb1f94884976a034064808f1b9af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44219 } } }
I20260927 15:00:52.586222 18533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9631cb1f94884976a034064808f1b9af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9631cb1f94884976a034064808f1b9af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9631cb1f94884976a034064808f1b9af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44219 } } }
I20260927 15:00:52.586366 18533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9631cb1f94884976a034064808f1b9af [sys.catalog]: This master's current role is: LEADER
I20260927 15:00:52.586591 18534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9631cb1f94884976a034064808f1b9af [sys.catalog]: This master's current role is: LEADER
I20260927 15:00:52.586896 18542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:00:52.587396 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:52.573196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55172 (local address 127.0.98.62:44219)
0927 15:00:52.573610 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:52.573628 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:52.573664 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:00:52.574351 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:52.574375 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:52.574786 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:52.575173 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:52.576010 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:52.577050 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:52.578226 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:52.578712 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:52.578874 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:52.581746 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:52.581752 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:52.581768 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:00:52.581801 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:52.583826 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:52.584382 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:52.584389 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:52.584396 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:52.584540 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:52.584823 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:52.584827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:52.584828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:52.585352 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:52.585564 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:52.585895 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 15:00:52.586228 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":117,"threads_started":1}
I20260927 15:00:52.590173 18542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:00:52.598379 18542 catalog_manager.cc:1347] Generated new cluster ID: afec037c38c84d63a600bebc336bf955
I20260927 15:00:52.598515 18542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:00:52.609375 18542 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:00:52.610502 18542 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:00:52.620483 18542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9631cb1f94884976a034064808f1b9af: Generated new TSK 0
I20260927 15:00:52.621237 18542 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 15:00:52.651932 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44219
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:00:52.800585 18553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:52.800880 18553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:52.800925 18553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:52.805074 18553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:52.805171 18553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:52.805234 18553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:00:52.810737 18553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:52.811924 18553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:52.812971 18553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:00:52.822773 18553 server_base.cc:1034] running on GCE node
W20260927 15:00:52.822717 18558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:52.822719 18559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:52.822717 18561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:52.823419 18553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:52.824070 18553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:52.825227 18553 hybrid_clock.cc:648] HybridClock initialized: now 1790521252825209 us; error 33 us; skew 500 ppm
Sep 27 15:00:52 dist-test-slave-pz0h krb5kdc[18446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:00:52.828107 18553 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 15:00:52.829139 18553 webserver.cc:460] Webserver started at http://127.0.98.1:35435/ using document root <none> and password file <none>
I20260927 15:00:52.829695 18553 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:52.829763 18553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:52.829960 18553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:00:52.832072 18553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-0/data/instance:
uuid: "15fc70ab55324f69bfa8bea2b8767a21"
format_stamp: "Formatted at 2026-09-27 15:00:52 on dist-test-slave-pz0h"
server_key: "38c1e5f9da6d41ea6ec839a01e7ea6c8"
server_key_iv: "132ccb790aceb56107f39caaa4efd5c6"
server_key_version: "encryptionkey@0"
I20260927 15:00:52.832631 18553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "15fc70ab55324f69bfa8bea2b8767a21"
format_stamp: "Formatted at 2026-09-27 15:00:52 on dist-test-slave-pz0h"
server_key: "38c1e5f9da6d41ea6ec839a01e7ea6c8"
server_key_iv: "132ccb790aceb56107f39caaa4efd5c6"
server_key_version: "encryptionkey@0"
I20260927 15:00:52.836619 18553 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:00:52.839296 18568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:52.840593 18553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:00:52.840741 18553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "15fc70ab55324f69bfa8bea2b8767a21"
format_stamp: "Formatted at 2026-09-27 15:00:52 on dist-test-slave-pz0h"
server_key: "38c1e5f9da6d41ea6ec839a01e7ea6c8"
server_key_iv: "132ccb790aceb56107f39caaa4efd5c6"
server_key_version: "encryptionkey@0"
I20260927 15:00:52.840855 18553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:52.885528 18553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:52.888820 18553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:52.889012 18553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:52.889648 18553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:00:52.890486 18553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:00:52.890534 18553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:52.890587 18553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:00:52.890614 18553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:52.899132 18553 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39583
I20260927 15:00:52.899161 18681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39583 every 8 connection(s)
I20260927 15:00:52.900068 18553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 15:00:52 dist-test-slave-pz0h krb5kdc[18446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:00:52.909902 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 18553
I20260927 15:00:52.910017 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:00:52.910339 392 external_mini_cluster.cc:1428] Setting key 12ebcfd3f0476bc044e2138a34548ce2
I20260927 15:00:52.913690 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:52.901955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33693 (local address 127.0.98.62:44219)
0927 15:00:52.902125 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:52.902129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:52.903008 (+ 879us) server_negotiation.cc:407] Connection header received
0927 15:00:52.904250 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:52.904253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:52.904303 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:52.904380 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:52.905564 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:52.906095 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:52.907298 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:52.907527 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:52.907609 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:52.910201 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:52.910206 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:52.910210 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:00:52.910242 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:52.911597 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:52.912170 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:52.912174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:52.912176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:52.912272 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:52.912572 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:52.912575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:52.912577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:52.912798 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:52.912883 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:52.913392 (+ 509us) server_negotiation.cc:299] Negotiation successful
0927 15:00:52.913524 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 15:00:52.914608 18684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:52.902365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44219 (local address 127.0.98.1:33693)
0927 15:00:52.902836 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:52.902894 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:52.904027 (+ 1133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:52.904539 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:52.904546 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:52.904914 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:00:52.905414 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:52.905427 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:52.906204 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:52.906206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:52.907192 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:52.907201 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:52.907698 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:52.907702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:52.907858 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:52.908443 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:00:52.908460 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:00:52.909977 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:00:52.911728 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:00:52.911735 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:00:52.911754 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:00:52.912069 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:00:52.912355 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:00:52.912357 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:00:52.912358 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:00:52.912493 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:00:52.912885 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:00:52.912890 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:00:52.913150 (+ 260us) client_negotiation.cc:769] Sending connection context
0927 15:00:52.913336 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 15:00:52.913695 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":109,"threads_started":1}
I20260927 15:00:52.915891 18682 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44219
I20260927 15:00:52.916141 18682 heartbeater.cc:464] Registering TS with master...
I20260927 15:00:52.916705 18682 heartbeater.cc:511] Master 127.0.98.62:44219 requested a full tablet report, sending...
I20260927 15:00:52.918148 18486 ts_manager.cc:194] Registered new tserver with Master: 15fc70ab55324f69bfa8bea2b8767a21 (127.0.98.1:39583)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 15:00:52.919387 18486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33693
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 15:00:52.937354 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44219
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:00:53.051151 18687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:53.051446 18687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:53.051491 18687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:53.055790 18687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:53.055866 18687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:53.055930 18687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:00:53.061517 18687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:53.062670 18687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:53.063800 18687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:00:53.073630 18692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:53.073670 18693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:53.073632 18695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:53.074023 18687 server_base.cc:1034] running on GCE node
I20260927 15:00:53.074522 18687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:53.075229 18687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:53.076434 18687 hybrid_clock.cc:648] HybridClock initialized: now 1790521253076417 us; error 38 us; skew 500 ppm
Sep 27 15:00:53 dist-test-slave-pz0h krb5kdc[18446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:00:53.079339 18687 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 15:00:53.080328 18687 webserver.cc:460] Webserver started at http://127.0.98.2:40015/ using document root <none> and password file <none>
I20260927 15:00:53.080900 18687 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:53.080968 18687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:53.081161 18687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:00:53.083473 18687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-1/data/instance:
uuid: "3fee17f6d4e44f20ae046cd5231c2730"
format_stamp: "Formatted at 2026-09-27 15:00:53 on dist-test-slave-pz0h"
server_key: "416e141d4c97d67d00ffd531df4a6bf6"
server_key_iv: "37b7ae07bd71f96e6c8a82a81b6f8969"
server_key_version: "encryptionkey@0"
I20260927 15:00:53.084101 18687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3fee17f6d4e44f20ae046cd5231c2730"
format_stamp: "Formatted at 2026-09-27 15:00:53 on dist-test-slave-pz0h"
server_key: "416e141d4c97d67d00ffd531df4a6bf6"
server_key_iv: "37b7ae07bd71f96e6c8a82a81b6f8969"
server_key_version: "encryptionkey@0"
I20260927 15:00:53.088606 18687 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:00:53.091609 18702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:53.093324 18687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:00:53.093468 18687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "3fee17f6d4e44f20ae046cd5231c2730"
format_stamp: "Formatted at 2026-09-27 15:00:53 on dist-test-slave-pz0h"
server_key: "416e141d4c97d67d00ffd531df4a6bf6"
server_key_iv: "37b7ae07bd71f96e6c8a82a81b6f8969"
server_key_version: "encryptionkey@0"
I20260927 15:00:53.093586 18687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:53.125200 18687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:53.128324 18687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:53.128518 18687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:53.129144 18687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:00:53.130018 18687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:00:53.130067 18687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:53.130121 18687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:00:53.130146 18687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:53.138711 18687 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35999
I20260927 15:00:53.138736 18815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35999 every 8 connection(s)
I20260927 15:00:53.139699 18687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:00:53.144187 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 18687
I20260927 15:00:53.144309 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:00:53.144603 392 external_mini_cluster.cc:1428] Setting key 6b443e3766bdfc572ad5ff1bf56041dc
Sep 27 15:00:53 dist-test-slave-pz0h krb5kdc[18446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 15:00:53.155802 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:53.141571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50227 (local address 127.0.98.62:44219)
0927 15:00:53.141747 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:53.141752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:53.142612 (+ 860us) server_negotiation.cc:407] Connection header received
0927 15:00:53.143909 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:53.143912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:53.143964 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:53.144039 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:53.145724 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.146414 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.148196 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.148542 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.148658 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:53.151678 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:53.151684 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:53.151687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:00:53.151726 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:53.153466 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:53.154111 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:53.154115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:53.154117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:53.154217 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:53.154569 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:53.154573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:53.154575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:53.154872 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:53.154975 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:53.155555 (+ 580us) server_negotiation.cc:299] Negotiation successful
0927 15:00:53.155671 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:00:53.156771 18818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:53.141976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44219 (local address 127.0.98.2:50227)
0927 15:00:53.142452 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:53.142496 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:53.143683 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:53.144256 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:53.144268 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:53.144824 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:00:53.145535 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:53.145556 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.146573 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.146577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:53.147987 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:53.148004 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.148713 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.148716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:53.148879 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:53.149531 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:00:53.149552 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:00:53.151349 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:00:53.153645 (+ 2296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:00:53.153652 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:00:53.153670 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:00:53.153980 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:00:53.154316 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:00:53.154318 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:00:53.154320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:00:53.154461 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:00:53.154967 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:00:53.154974 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:00:53.155283 (+ 309us) client_negotiation.cc:769] Sending connection context
0927 15:00:53.155471 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 15:00:53.155772 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":128,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 15:00:53.158067 18816 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44219
I20260927 15:00:53.158394 18816 heartbeater.cc:464] Registering TS with master...
I20260927 15:00:53.159054 18816 heartbeater.cc:511] Master 127.0.98.62:44219 requested a full tablet report, sending...
I20260927 15:00:53.160323 18486 ts_manager.cc:194] Registered new tserver with Master: 3fee17f6d4e44f20ae046cd5231c2730 (127.0.98.2:35999)
I20260927 15:00:53.160903 18486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50227
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 15:00:53.171929 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44219
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:00:53.288103 18821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:53.288429 18821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:53.288498 18821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:53.293217 18821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:53.293325 18821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:53.293418 18821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:00:53.299708 18821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:53.300853 18821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:53.301875 18821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:00:53.311566 18827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:53.311537 18829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:53.311576 18826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:53.311801 18821 server_base.cc:1034] running on GCE node
I20260927 15:00:53.312358 18821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:53.312999 18821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:53.314183 18821 hybrid_clock.cc:648] HybridClock initialized: now 1790521253314160 us; error 38 us; skew 500 ppm
Sep 27 15:00:53 dist-test-slave-pz0h krb5kdc[18446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:00:53.317102 18821 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 15:00:53.317989 18821 webserver.cc:460] Webserver started at http://127.0.98.3:45415/ using document root <none> and password file <none>
I20260927 15:00:53.318517 18821 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:53.318576 18821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:53.318750 18821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:00:53.320868 18821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-2/data/instance:
uuid: "fe8d160475fa4700ba70dff3a6fac4db"
format_stamp: "Formatted at 2026-09-27 15:00:53 on dist-test-slave-pz0h"
server_key: "441c5009dcce6f7d2f7de10862536d3f"
server_key_iv: "8f02bbdff9f2f73354d0b9ab19cefbd4"
server_key_version: "encryptionkey@0"
I20260927 15:00:53.321384 18821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fe8d160475fa4700ba70dff3a6fac4db"
format_stamp: "Formatted at 2026-09-27 15:00:53 on dist-test-slave-pz0h"
server_key: "441c5009dcce6f7d2f7de10862536d3f"
server_key_iv: "8f02bbdff9f2f73354d0b9ab19cefbd4"
server_key_version: "encryptionkey@0"
I20260927 15:00:53.325491 18821 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:00:53.328699 18836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:53.330119 18821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:00:53.330251 18821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "fe8d160475fa4700ba70dff3a6fac4db"
format_stamp: "Formatted at 2026-09-27 15:00:53 on dist-test-slave-pz0h"
server_key: "441c5009dcce6f7d2f7de10862536d3f"
server_key_iv: "8f02bbdff9f2f73354d0b9ab19cefbd4"
server_key_version: "encryptionkey@0"
I20260927 15:00:53.330380 18821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:53.355587 18821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:53.359130 18821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:53.359349 18821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:53.359983 18821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:00:53.360831 18821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:00:53.360888 18821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:53.360927 18821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:00:53.360955 18821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:53.369451 18821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46207
I20260927 15:00:53.369477 18949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46207 every 8 connection(s)
I20260927 15:00:53.370352 18821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:00:53.379084 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 18821
I20260927 15:00:53.379241 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-2/wal/instance
I20260927 15:00:53.379518 392 external_mini_cluster.cc:1428] Setting key 6e367a23f6e445570557cb2248794715
Sep 27 15:00:53 dist-test-slave-pz0h krb5kdc[18446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:00:53.384938 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:53.372275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37129 (local address 127.0.98.62:44219)
0927 15:00:53.372431 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:53.372435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:53.373274 (+ 839us) server_negotiation.cc:407] Connection header received
0927 15:00:53.374540 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:53.374543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:53.374603 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:53.374681 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:53.375962 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.376637 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.378016 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.378258 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.378351 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:53.381226 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:53.381230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:53.381233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:00:53.381263 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:53.382610 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:53.383214 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:53.383217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:53.383218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:53.383315 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:53.383814 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:53.383817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:53.383818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:53.384021 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:53.384102 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:53.384653 (+ 551us) server_negotiation.cc:299] Negotiation successful
0927 15:00:53.384812 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 15:00:53.385834 18952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:53.372651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44219 (local address 127.0.98.3:37129)
0927 15:00:53.373117 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:53.373161 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:53.374319 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:53.374895 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:53.374902 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:53.375273 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:00:53.375798 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:53.375814 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.376800 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.376804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:53.377867 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:53.377879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.378394 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.378396 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:53.378541 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:53.379200 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:00:53.379219 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:00:53.381058 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:00:53.382746 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:00:53.382752 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:00:53.382766 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:00:53.383106 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:00:53.383504 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:00:53.383507 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:00:53.383509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:00:53.383676 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:00:53.384130 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:00:53.384135 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:00:53.384418 (+ 283us) client_negotiation.cc:769] Sending connection context
0927 15:00:53.384581 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 15:00:53.384942 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":106,"threads_started":1}
I20260927 15:00:53.387085 18950 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44219
I20260927 15:00:53.387327 18950 heartbeater.cc:464] Registering TS with master...
I20260927 15:00:53.387840 18950 heartbeater.cc:511] Master 127.0.98.62:44219 requested a full tablet report, sending...
I20260927 15:00:53.388880 18486 ts_manager.cc:194] Registered new tserver with Master: fe8d160475fa4700ba70dff3a6fac4db (127.0.98.3:46207)
I20260927 15:00:53.389454 18486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37129
I20260927 15:00:53.393872 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:00:53.403884 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:53.395832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55180 (local address 127.0.98.62:44219)
0927 15:00:53.395966 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:53.395970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:53.396095 (+ 125us) server_negotiation.cc:407] Connection header received
0927 15:00:53.396239 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:53.396242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:53.396293 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:53.396373 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:53.397072 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.397615 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.398583 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.398877 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.398964 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:53.400631 (+ 1667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:53.400635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:53.400637 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:00:53.400674 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:53.402054 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:53.402615 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:53.402618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:53.402620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:53.402714 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:53.403024 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:53.403026 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:53.403028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:53.403246 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:53.403370 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:53.403621 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 15:00:53.403714 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:00:53.407994 18486 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:00:53.411329 18486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:00:53.428928 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:53.422392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39583 (local address 127.0.0.1:36082)
0927 15:00:53.422891 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:53.422910 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:53.423137 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:53.424057 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:53.424063 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:53.424086 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:53.424234 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:53.424242 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.425755 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.425759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:53.427327 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:53.427338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.428482 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.428486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:53.428578 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:53.428602 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:00:53.428678 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 15:00:53.428757 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":1,"thread_start_us":106,"threads_started":1}
I20260927 15:00:53.428972 18964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:53.422561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36082 (local address 127.0.98.1:39583)
0927 15:00:53.423341 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:53.423349 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:53.423374 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:00:53.423455 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:53.423461 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:53.423714 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:53.423879 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:53.424368 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.425565 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.427473 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.428249 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.428363 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:53.428509 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:53.428729 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 15:00:53.428832 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":94,"threads_started":1}
I20260927 15:00:53.429088 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:53.423336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45344 (local address 127.0.98.3:46207)
0927 15:00:53.424089 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:53.424114 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:53.424135 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:00:53.424207 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:53.424212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:53.424408 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:53.424545 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:53.424930 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.425917 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.427258 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.428138 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.428243 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:53.428408 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:53.428853 (+ 445us) server_negotiation.cc:299] Negotiation successful
0927 15:00:53.428947 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":153,"threads_started":1}
I20260927 15:00:53.429342 18965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:53.422912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46207 (local address 127.0.0.1:45344)
0927 15:00:53.423749 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:53.423763 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:53.423850 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:53.424712 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:53.424715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:53.424733 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:53.424809 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:53.424815 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.426047 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.426051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:53.427104 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:53.427119 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.428289 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.428293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:53.428413 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:53.428453 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 15:00:53.428564 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 15:00:53.428686 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":83,"thread_start_us":103,"threads_started":1}
I20260927 15:00:53.430312 18961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:53.422136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35999 (local address 127.0.0.1:50750)
0927 15:00:53.422853 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:53.422909 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:53.423045 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:53.424893 (+ 1848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:53.424896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:53.424914 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:53.425002 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:53.425008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.427167 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.427171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:53.428264 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:53.428274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.429888 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.429892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:53.430008 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:53.430032 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:00:53.430108 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 15:00:53.430191 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":96,"threads_started":1}
I20260927 15:00:53.430501 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:53.422316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50750 (local address 127.0.98.2:35999)
0927 15:00:53.424436 (+ 2120us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:53.424451 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:53.424474 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:00:53.424548 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:53.424553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:53.424730 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:53.425359 (+ 629us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:53.425610 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.427011 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.428412 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.429764 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.429877 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:53.430015 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:53.430171 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 15:00:53.430352 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1999,"thread_start_us":117,"threads_started":1}
I20260927 15:00:53.433112 18616 tablet_service.cc:1467] Processing CreateTablet for tablet 4536105e29a1494ba04ec339c095a1b2 (DEFAULT_TABLE table=test-table [id=dab628407dcc4922a20e4b2e5ec0ce0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:00:53.433112 18884 tablet_service.cc:1467] Processing CreateTablet for tablet 4536105e29a1494ba04ec339c095a1b2 (DEFAULT_TABLE table=test-table [id=dab628407dcc4922a20e4b2e5ec0ce0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:00:53.433396 18750 tablet_service.cc:1467] Processing CreateTablet for tablet 4536105e29a1494ba04ec339c095a1b2 (DEFAULT_TABLE table=test-table [id=dab628407dcc4922a20e4b2e5ec0ce0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:00:53.434360 18884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4536105e29a1494ba04ec339c095a1b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:00:53.434360 18616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4536105e29a1494ba04ec339c095a1b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:00:53.434357 18750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4536105e29a1494ba04ec339c095a1b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:00:53.441563 18967 tablet_bootstrap.cc:492] T 4536105e29a1494ba04ec339c095a1b2 P 3fee17f6d4e44f20ae046cd5231c2730: Bootstrap starting.
I20260927 15:00:53.442723 18969 tablet_bootstrap.cc:492] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db: Bootstrap starting.
I20260927 15:00:53.442834 18968 tablet_bootstrap.cc:492] T 4536105e29a1494ba04ec339c095a1b2 P 15fc70ab55324f69bfa8bea2b8767a21: Bootstrap starting.
I20260927 15:00:53.444613 18969 tablet_bootstrap.cc:654] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db: Neither blocks nor log segments found. Creating new log.
I20260927 15:00:53.444679 18967 tablet_bootstrap.cc:654] T 4536105e29a1494ba04ec339c095a1b2 P 3fee17f6d4e44f20ae046cd5231c2730: Neither blocks nor log segments found. Creating new log.
I20260927 15:00:53.444974 18968 tablet_bootstrap.cc:654] T 4536105e29a1494ba04ec339c095a1b2 P 15fc70ab55324f69bfa8bea2b8767a21: Neither blocks nor log segments found. Creating new log.
I20260927 15:00:53.445649 18969 log.cc:824] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db: Log is configured to *not* fsync() on all Append() calls
I20260927 15:00:53.445663 18967 log.cc:824] T 4536105e29a1494ba04ec339c095a1b2 P 3fee17f6d4e44f20ae046cd5231c2730: Log is configured to *not* fsync() on all Append() calls
I20260927 15:00:53.445706 18968 log.cc:824] T 4536105e29a1494ba04ec339c095a1b2 P 15fc70ab55324f69bfa8bea2b8767a21: Log is configured to *not* fsync() on all Append() calls
I20260927 15:00:53.447945 18969 tablet_bootstrap.cc:492] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db: No bootstrap required, opened a new log
I20260927 15:00:53.447968 18968 tablet_bootstrap.cc:492] T 4536105e29a1494ba04ec339c095a1b2 P 15fc70ab55324f69bfa8bea2b8767a21: No bootstrap required, opened a new log
I20260927 15:00:53.448069 18967 tablet_bootstrap.cc:492] T 4536105e29a1494ba04ec339c095a1b2 P 3fee17f6d4e44f20ae046cd5231c2730: No bootstrap required, opened a new log
I20260927 15:00:53.448170 18968 ts_tablet_manager.cc:1397] T 4536105e29a1494ba04ec339c095a1b2 P 15fc70ab55324f69bfa8bea2b8767a21: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:00:53.448171 18969 ts_tablet_manager.cc:1397] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260927 15:00:53.448223 18967 ts_tablet_manager.cc:1397] T 4536105e29a1494ba04ec339c095a1b2 P 3fee17f6d4e44f20ae046cd5231c2730: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 15:00:53.452150 18967 raft_consensus.cc:348] T 4536105e29a1494ba04ec339c095a1b2 P 3fee17f6d4e44f20ae046cd5231c2730 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fee17f6d4e44f20ae046cd5231c2730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35999 } } peers { permanent_uuid: "15fc70ab55324f69bfa8bea2b8767a21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } } peers { permanent_uuid: "fe8d160475fa4700ba70dff3a6fac4db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46207 } }
I20260927 15:00:53.452299 18969 raft_consensus.cc:348] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fee17f6d4e44f20ae046cd5231c2730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35999 } } peers { permanent_uuid: "15fc70ab55324f69bfa8bea2b8767a21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } } peers { permanent_uuid: "fe8d160475fa4700ba70dff3a6fac4db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46207 } }
I20260927 15:00:53.452523 18967 raft_consensus.cc:374] T 4536105e29a1494ba04ec339c095a1b2 P 3fee17f6d4e44f20ae046cd5231c2730 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:00:53.452581 18967 raft_consensus.cc:729] T 4536105e29a1494ba04ec339c095a1b2 P 3fee17f6d4e44f20ae046cd5231c2730 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fee17f6d4e44f20ae046cd5231c2730, State: Initialized, Role: FOLLOWER
I20260927 15:00:53.452607 18969 raft_consensus.cc:374] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:00:53.452657 18969 raft_consensus.cc:729] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe8d160475fa4700ba70dff3a6fac4db, State: Initialized, Role: FOLLOWER
I20260927 15:00:53.452522 18968 raft_consensus.cc:348] T 4536105e29a1494ba04ec339c095a1b2 P 15fc70ab55324f69bfa8bea2b8767a21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fee17f6d4e44f20ae046cd5231c2730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35999 } } peers { permanent_uuid: "15fc70ab55324f69bfa8bea2b8767a21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } } peers { permanent_uuid: "fe8d160475fa4700ba70dff3a6fac4db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46207 } }
I20260927 15:00:53.452750 18968 raft_consensus.cc:374] T 4536105e29a1494ba04ec339c095a1b2 P 15fc70ab55324f69bfa8bea2b8767a21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:00:53.452824 18968 raft_consensus.cc:729] T 4536105e29a1494ba04ec339c095a1b2 P 15fc70ab55324f69bfa8bea2b8767a21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15fc70ab55324f69bfa8bea2b8767a21, State: Initialized, Role: FOLLOWER
I20260927 15:00:53.453096 18967 consensus_queue.cc:260] T 4536105e29a1494ba04ec339c095a1b2 P 3fee17f6d4e44f20ae046cd5231c2730 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fee17f6d4e44f20ae046cd5231c2730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35999 } } peers { permanent_uuid: "15fc70ab55324f69bfa8bea2b8767a21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } } peers { permanent_uuid: "fe8d160475fa4700ba70dff3a6fac4db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46207 } }
I20260927 15:00:53.453167 18968 consensus_queue.cc:260] T 4536105e29a1494ba04ec339c095a1b2 P 15fc70ab55324f69bfa8bea2b8767a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fee17f6d4e44f20ae046cd5231c2730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35999 } } peers { permanent_uuid: "15fc70ab55324f69bfa8bea2b8767a21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } } peers { permanent_uuid: "fe8d160475fa4700ba70dff3a6fac4db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46207 } }
I20260927 15:00:53.453219 18969 consensus_queue.cc:260] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fee17f6d4e44f20ae046cd5231c2730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35999 } } peers { permanent_uuid: "15fc70ab55324f69bfa8bea2b8767a21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } } peers { permanent_uuid: "fe8d160475fa4700ba70dff3a6fac4db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46207 } }
I20260927 15:00:53.454212 18968 ts_tablet_manager.cc:1428] T 4536105e29a1494ba04ec339c095a1b2 P 15fc70ab55324f69bfa8bea2b8767a21: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 15:00:53.454212 18967 ts_tablet_manager.cc:1428] T 4536105e29a1494ba04ec339c095a1b2 P 3fee17f6d4e44f20ae046cd5231c2730: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:00:53.454213 18969 ts_tablet_manager.cc:1428] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:00:53.454609 18816 heartbeater.cc:503] Master 127.0.98.62:44219 was elected leader, sending a full tablet report...
I20260927 15:00:53.454747 18682 heartbeater.cc:503] Master 127.0.98.62:44219 was elected leader, sending a full tablet report...
I20260927 15:00:53.455648 18950 heartbeater.cc:503] Master 127.0.98.62:44219 was elected leader, sending a full tablet report...
I20260927 15:00:53.526530 18975 raft_consensus.cc:482] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:00:53.526778 18975 raft_consensus.cc:504] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fee17f6d4e44f20ae046cd5231c2730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35999 } } peers { permanent_uuid: "15fc70ab55324f69bfa8bea2b8767a21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } } peers { permanent_uuid: "fe8d160475fa4700ba70dff3a6fac4db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46207 } }
I20260927 15:00:53.528050 18975 leader_election.cc:288] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fee17f6d4e44f20ae046cd5231c2730 (127.0.98.2:35999), 15fc70ab55324f69bfa8bea2b8767a21 (127.0.98.1:39583)
I20260927 15:00:53.532400 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:53.528759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34247 (local address 127.0.98.2:35999)
0927 15:00:53.528887 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:53.528892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:53.528904 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:00:53.528951 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:53.528954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:53.528992 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:53.529051 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:53.529391 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.529898 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.531124 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.531745 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.531846 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:53.531908 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:53.532220 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 15:00:53.532283 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 15:00:53.532462 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:53.528324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35999 (local address 127.0.98.3:34247)
0927 15:00:53.528612 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:53.528625 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:53.528709 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:53.529095 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:53.529098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:53.529119 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:53.529272 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:53.529279 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.530002 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.530005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:53.531005 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:53.531015 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.531972 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.531976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:53.532083 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:53.532112 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 15:00:53.532214 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 15:00:53.532292 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":23,"thread_start_us":109,"threads_started":1}
I20260927 15:00:53.533231 18952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:53.528299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39583 (local address 127.0.98.3:33383)
0927 15:00:53.528452 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:53.528469 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:53.528567 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:53.529150 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:53.529153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:53.529170 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:53.529274 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:53.529279 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.530643 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.530647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:53.531647 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:53.531658 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.532895 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.532898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:53.532976 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:53.532996 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 15:00:53.533060 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 15:00:53.533128 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":1}
I20260927 15:00:53.533224 18770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4536105e29a1494ba04ec339c095a1b2" candidate_uuid: "fe8d160475fa4700ba70dff3a6fac4db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fee17f6d4e44f20ae046cd5231c2730" is_pre_election: true
I20260927 15:00:53.533355 18964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:53.528431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33383 (local address 127.0.98.1:39583)
0927 15:00:53.528542 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:53.528547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:53.528565 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:00:53.528668 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:53.528671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:53.528722 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:53.528797 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:53.530025 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.530544 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.531948 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.532605 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.532722 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:53.532785 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:53.533119 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 15:00:53.533238 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 15:00:53.533607 18770 raft_consensus.cc:2454] T 4536105e29a1494ba04ec339c095a1b2 P 3fee17f6d4e44f20ae046cd5231c2730 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe8d160475fa4700ba70dff3a6fac4db in term 0.
I20260927 15:00:53.534003 18636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4536105e29a1494ba04ec339c095a1b2" candidate_uuid: "fe8d160475fa4700ba70dff3a6fac4db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15fc70ab55324f69bfa8bea2b8767a21" is_pre_election: true
I20260927 15:00:53.534137 18840 leader_election.cc:302] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fee17f6d4e44f20ae046cd5231c2730, fe8d160475fa4700ba70dff3a6fac4db; no voters:
I20260927 15:00:53.534299 18636 raft_consensus.cc:2454] T 4536105e29a1494ba04ec339c095a1b2 P 15fc70ab55324f69bfa8bea2b8767a21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe8d160475fa4700ba70dff3a6fac4db in term 0.
I20260927 15:00:53.534440 18975 raft_consensus.cc:2781] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:00:53.534533 18975 raft_consensus.cc:482] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:00:53.534575 18975 raft_consensus.cc:3037] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:00:53.535815 18975 raft_consensus.cc:504] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fee17f6d4e44f20ae046cd5231c2730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35999 } } peers { permanent_uuid: "15fc70ab55324f69bfa8bea2b8767a21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } } peers { permanent_uuid: "fe8d160475fa4700ba70dff3a6fac4db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46207 } }
I20260927 15:00:53.536154 18975 leader_election.cc:288] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db [CANDIDATE]: Term 1 election: Requested vote from peers 3fee17f6d4e44f20ae046cd5231c2730 (127.0.98.2:35999), 15fc70ab55324f69bfa8bea2b8767a21 (127.0.98.1:39583)
I20260927 15:00:53.536504 18770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4536105e29a1494ba04ec339c095a1b2" candidate_uuid: "fe8d160475fa4700ba70dff3a6fac4db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fee17f6d4e44f20ae046cd5231c2730"
I20260927 15:00:53.536531 18636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4536105e29a1494ba04ec339c095a1b2" candidate_uuid: "fe8d160475fa4700ba70dff3a6fac4db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15fc70ab55324f69bfa8bea2b8767a21"
I20260927 15:00:53.536636 18770 raft_consensus.cc:3037] T 4536105e29a1494ba04ec339c095a1b2 P 3fee17f6d4e44f20ae046cd5231c2730 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:00:53.536643 18636 raft_consensus.cc:3037] T 4536105e29a1494ba04ec339c095a1b2 P 15fc70ab55324f69bfa8bea2b8767a21 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:00:53.537683 18770 raft_consensus.cc:2454] T 4536105e29a1494ba04ec339c095a1b2 P 3fee17f6d4e44f20ae046cd5231c2730 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe8d160475fa4700ba70dff3a6fac4db in term 1.
I20260927 15:00:53.537699 18636 raft_consensus.cc:2454] T 4536105e29a1494ba04ec339c095a1b2 P 15fc70ab55324f69bfa8bea2b8767a21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe8d160475fa4700ba70dff3a6fac4db in term 1.
I20260927 15:00:53.538013 18837 leader_election.cc:302] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15fc70ab55324f69bfa8bea2b8767a21, fe8d160475fa4700ba70dff3a6fac4db; no voters:
I20260927 15:00:53.538173 18975 raft_consensus.cc:2781] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:00:53.538390 18975 raft_consensus.cc:686] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db [term 1 LEADER]: Becoming Leader. State: Replica: fe8d160475fa4700ba70dff3a6fac4db, State: Running, Role: LEADER
I20260927 15:00:53.538719 18975 consensus_queue.cc:237] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fee17f6d4e44f20ae046cd5231c2730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35999 } } peers { permanent_uuid: "15fc70ab55324f69bfa8bea2b8767a21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } } peers { permanent_uuid: "fe8d160475fa4700ba70dff3a6fac4db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46207 } }
I20260927 15:00:53.542084 18491 catalog_manager.cc:5495] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db reported cstate change: term changed from 0 to 1, leader changed from <none> to fe8d160475fa4700ba70dff3a6fac4db (127.0.98.3). New cstate: current_term: 1 leader_uuid: "fe8d160475fa4700ba70dff3a6fac4db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fee17f6d4e44f20ae046cd5231c2730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15fc70ab55324f69bfa8bea2b8767a21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe8d160475fa4700ba70dff3a6fac4db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46207 } health_report { overall_health: HEALTHY } } }
I20260927 15:00:53.555986 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:53.552443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45360 (local address 127.0.98.3:46207)
0927 15:00:53.552592 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:53.552596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:53.552613 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:00:53.552691 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:53.552693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:53.552748 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:53.552842 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:00:53.553080 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.553583 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.554388 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.554594 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.554679 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:53.555026 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:00:53.555453 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:00:53.555599 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:53.555701 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 15:00:53.555743 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 15:00:53.562739 18770 raft_consensus.cc:1261] T 4536105e29a1494ba04ec339c095a1b2 P 3fee17f6d4e44f20ae046cd5231c2730 [term 1 FOLLOWER]: Refusing update from remote peer fe8d160475fa4700ba70dff3a6fac4db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:00:53.562764 18636 raft_consensus.cc:1261] T 4536105e29a1494ba04ec339c095a1b2 P 15fc70ab55324f69bfa8bea2b8767a21 [term 1 FOLLOWER]: Refusing update from remote peer fe8d160475fa4700ba70dff3a6fac4db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:00:53.563550 18977 consensus_queue.cc:1035] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db [LEADER]: Connected to new peer: Peer: permanent_uuid: "15fc70ab55324f69bfa8bea2b8767a21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:00:53.563768 18975 consensus_queue.cc:1035] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fee17f6d4e44f20ae046cd5231c2730" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:00:53.575729 18982 mvcc.cc:205] Tried to move back new op lower bound from 7333975054585004032 to 7333975054497382400. Current Snapshot: MvccSnapshot[applied={T|T < 7333975054585004032}]
I20260927 15:00:53.583503 18491 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:00:53.583691 18491 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:00:53.586205 18491 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9631cb1f94884976a034064808f1b9af: Sending DeleteTablet for 3 replicas of tablet 4536105e29a1494ba04ec339c095a1b2
I20260927 15:00:53.587177 18616 tablet_service.cc:1514] Processing DeleteTablet for tablet 4536105e29a1494ba04ec339c095a1b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:00:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36082
I20260927 15:00:53.587363 18750 tablet_service.cc:1514] Processing DeleteTablet for tablet 4536105e29a1494ba04ec339c095a1b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:00:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50750
I20260927 15:00:53.587847 18990 tablet_replica.cc:330] T 4536105e29a1494ba04ec339c095a1b2 P 15fc70ab55324f69bfa8bea2b8767a21: stopping tablet replica
I20260927 15:00:53.587869 18884 tablet_service.cc:1514] Processing DeleteTablet for tablet 4536105e29a1494ba04ec339c095a1b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:00:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45344
I20260927 15:00:53.588223 18990 raft_consensus.cc:2229] T 4536105e29a1494ba04ec339c095a1b2 P 15fc70ab55324f69bfa8bea2b8767a21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:00:53.588340 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 18553
I20260927 15:00:53.588346 18992 tablet_replica.cc:330] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db: stopping tablet replica
I20260927 15:00:53.588699 18992 raft_consensus.cc:2229] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:00:53.589282 18992 raft_consensus.cc:2258] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:00:53.588600 18990 raft_consensus.cc:2258] T 4536105e29a1494ba04ec339c095a1b2 P 15fc70ab55324f69bfa8bea2b8767a21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:00:53.591006 18992 ts_tablet_manager.cc:1905] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:00:53.593961 18992 ts_tablet_manager.cc:1918] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:00:53.594154 18992 log.cc:1196] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790521004237705-392-0/minicluster-data/ts-2/wal/wals/4536105e29a1494ba04ec339c095a1b2
I20260927 15:00:53.594576 18992 ts_tablet_manager.cc:1939] T 4536105e29a1494ba04ec339c095a1b2 P fe8d160475fa4700ba70dff3a6fac4db: Deleting consensus metadata
I20260927 15:00:53.595301 18991 tablet_replica.cc:330] T 4536105e29a1494ba04ec339c095a1b2 P 3fee17f6d4e44f20ae046cd5231c2730: stopping tablet replica
I20260927 15:00:53.595662 18991 raft_consensus.cc:2229] T 4536105e29a1494ba04ec339c095a1b2 P 3fee17f6d4e44f20ae046cd5231c2730 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:00:53.595997 18991 raft_consensus.cc:2258] T 4536105e29a1494ba04ec339c095a1b2 P 3fee17f6d4e44f20ae046cd5231c2730 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:00:53.596140 18478 catalog_manager.cc:4841] TS fe8d160475fa4700ba70dff3a6fac4db (127.0.98.3:46207): tablet 4536105e29a1494ba04ec339c095a1b2 (table test-table [id=dab628407dcc4922a20e4b2e5ec0ce0a]) successfully deleted
W20260927 15:00:53.596930 18475 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39583 (error 108)
I20260927 15:00:53.597605 18991 ts_tablet_manager.cc:1905] T 4536105e29a1494ba04ec339c095a1b2 P 3fee17f6d4e44f20ae046cd5231c2730: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:00:53.597666 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 18687
I20260927 15:00:53.599161 18965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:53.598399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39583 (local address 127.0.0.1:36094)
0927 15:00:53.598972 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:53.599044 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486}
W20260927 15:00:53.599412 18475 catalog_manager.cc:4584] TS 15fc70ab55324f69bfa8bea2b8767a21 (127.0.98.1:39583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4536105e29a1494ba04ec339c095a1b2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39583: connect: Connection refused (error 111)
I20260927 15:00:53.604578 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 18821
I20260927 15:00:53.604815 18965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:53.604509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35999 (local address 127.0.0.1:50762)
0927 15:00:53.604651 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:53.604720 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 15:00:53.609988 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 18459
2026-09-27T15:00:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2337 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T15:00:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:00:53Z Disabled control of system clock
I20260927 15:00:53.639334 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33743
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:00:53.751171 18997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:53.751458 18997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:53.751494 18997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:53.755692 18997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:00:53.755760 18997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:53.755774 18997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:53.755790 18997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:00:53.755805 18997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:00:53.760776 18997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33743
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:53.761933 18997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:53.763433 18997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:00:53.773638 19003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:53.773638 19002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:53.774147 19005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:53.774369 18997 server_base.cc:1034] running on GCE node
I20260927 15:00:53.775070 18997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:53.776543 18997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:53.777746 18997 hybrid_clock.cc:648] HybridClock initialized: now 1790521253777720 us; error 44 us; skew 500 ppm
I20260927 15:00:53.779675 18997 webserver.cc:460] Webserver started at http://127.0.98.62:33619/ using document root <none> and password file <none>
I20260927 15:00:53.780223 18997 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:53.780284 18997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:53.780453 18997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:00:53.782522 18997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "bdedab0848e6442da46b2617381213fb"
format_stamp: "Formatted at 2026-09-27 15:00:53 on dist-test-slave-pz0h"
server_key: "b0df0fcef850e688b65b72c0470d3156"
server_key_iv: "e5160b8d2f7ec13c39d6a8c8e282683f"
server_key_version: "encryptionkey@0"
I20260927 15:00:53.783135 18997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "bdedab0848e6442da46b2617381213fb"
format_stamp: "Formatted at 2026-09-27 15:00:53 on dist-test-slave-pz0h"
server_key: "b0df0fcef850e688b65b72c0470d3156"
server_key_iv: "e5160b8d2f7ec13c39d6a8c8e282683f"
server_key_version: "encryptionkey@0"
I20260927 15:00:53.787138 18997 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:00:53.789692 19011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:53.791164 18997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:00:53.791291 18997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "bdedab0848e6442da46b2617381213fb"
format_stamp: "Formatted at 2026-09-27 15:00:53 on dist-test-slave-pz0h"
server_key: "b0df0fcef850e688b65b72c0470d3156"
server_key_iv: "e5160b8d2f7ec13c39d6a8c8e282683f"
server_key_version: "encryptionkey@0"
I20260927 15:00:53.791396 18997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:53.813897 18997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:53.814656 18997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:53.814883 18997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:53.821974 18997 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33743
I20260927 15:00:53.821991 19063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33743 every 8 connection(s)
I20260927 15:00:53.823055 18997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:00:53.825599 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 18997
I20260927 15:00:53.825714 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 15:00:53.825976 392 external_mini_cluster.cc:1428] Setting key 9af525e4d27acca29c7158ea6d271b7c
I20260927 15:00:53.826413 19064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:00:53.833818 19064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdedab0848e6442da46b2617381213fb: Bootstrap starting.
I20260927 15:00:53.836212 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:53.827468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60356 (local address 127.0.98.62:33743)
0927 15:00:53.828223 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:53.828239 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:53.828275 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:00:53.828963 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:53.828980 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:53.829395 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:53.829682 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:53.830030 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.831056 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.832224 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:53.832953 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:53.833103 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:53.834129 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:53.834135 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:53.834152 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:00:53.834641 (+ 489us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:53.834712 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:00:53.834739 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:53.834860 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:53.835001 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 15:00:53.835294 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":221,"threads_started":1}
I20260927 15:00:53.836556 19064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdedab0848e6442da46b2617381213fb: Neither blocks nor log segments found. Creating new log.
I20260927 15:00:53.837458 19064 log.cc:824] T 00000000000000000000000000000000 P bdedab0848e6442da46b2617381213fb: Log is configured to *not* fsync() on all Append() calls
I20260927 15:00:53.839761 19064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdedab0848e6442da46b2617381213fb: No bootstrap required, opened a new log
I20260927 15:00:53.843540 19064 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdedab0848e6442da46b2617381213fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdedab0848e6442da46b2617381213fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33743 } }
I20260927 15:00:53.843813 19064 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdedab0848e6442da46b2617381213fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:00:53.843848 19064 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdedab0848e6442da46b2617381213fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdedab0848e6442da46b2617381213fb, State: Initialized, Role: FOLLOWER
I20260927 15:00:53.844297 19064 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdedab0848e6442da46b2617381213fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdedab0848e6442da46b2617381213fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33743 } }
I20260927 15:00:53.844429 19064 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdedab0848e6442da46b2617381213fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:00:53.844493 19064 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdedab0848e6442da46b2617381213fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:00:53.844588 19064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdedab0848e6442da46b2617381213fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:00:53.845558 19064 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdedab0848e6442da46b2617381213fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdedab0848e6442da46b2617381213fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33743 } }
I20260927 15:00:53.845899 19064 leader_election.cc:302] T 00000000000000000000000000000000 P bdedab0848e6442da46b2617381213fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdedab0848e6442da46b2617381213fb; no voters:
I20260927 15:00:53.846226 19064 leader_election.cc:288] T 00000000000000000000000000000000 P bdedab0848e6442da46b2617381213fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:00:53.846371 19069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdedab0848e6442da46b2617381213fb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:00:53.846585 19069 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdedab0848e6442da46b2617381213fb [term 1 LEADER]: Becoming Leader. State: Replica: bdedab0848e6442da46b2617381213fb, State: Running, Role: LEADER
I20260927 15:00:53.846923 19069 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdedab0848e6442da46b2617381213fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdedab0848e6442da46b2617381213fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33743 } }
I20260927 15:00:53.847560 19064 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdedab0848e6442da46b2617381213fb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:00:53.848670 19071 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdedab0848e6442da46b2617381213fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdedab0848e6442da46b2617381213fb. Latest consensus state: current_term: 1 leader_uuid: "bdedab0848e6442da46b2617381213fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdedab0848e6442da46b2617381213fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33743 } } }
I20260927 15:00:53.848842 19071 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdedab0848e6442da46b2617381213fb [sys.catalog]: This master's current role is: LEADER
I20260927 15:00:53.848670 19070 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdedab0848e6442da46b2617381213fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdedab0848e6442da46b2617381213fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdedab0848e6442da46b2617381213fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33743 } } }
I20260927 15:00:53.849058 19070 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdedab0848e6442da46b2617381213fb [sys.catalog]: This master's current role is: LEADER
I20260927 15:00:53.849237 19078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:00:53.853333 19078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:00:53.861593 19078 catalog_manager.cc:1347] Generated new cluster ID: ee8a2681c3d24628a34d51377afd78aa
I20260927 15:00:53.861769 19078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:00:53.876847 19078 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:00:53.878176 19078 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:00:53.890360 19078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdedab0848e6442da46b2617381213fb: Generated new TSK 0
I20260927 15:00:53.891180 19078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:00:53.901767 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33743
--builtin_ntp_servers=127.0.98.20:40801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:00:54.014109 19088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:54.014415 19088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:54.014454 19088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:54.018635 19088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:54.018707 19088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:54.018769 19088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:00:54.023747 19088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:54.024868 19088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:54.026252 19088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:00:54.036023 19096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:54.036068 19088 server_base.cc:1034] running on GCE node
W20260927 15:00:54.036006 19093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:54.036006 19094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:54.036652 19088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:54.037518 19088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:54.038714 19088 hybrid_clock.cc:648] HybridClock initialized: now 1790521254038687 us; error 40 us; skew 500 ppm
I20260927 15:00:54.040807 19088 webserver.cc:460] Webserver started at http://127.0.98.1:45321/ using document root <none> and password file <none>
I20260927 15:00:54.041397 19088 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:54.041464 19088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:54.041661 19088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:00:54.043820 19088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-0/data/instance:
uuid: "978b92b3c92747f28171de4cc729c29c"
format_stamp: "Formatted at 2026-09-27 15:00:54 on dist-test-slave-pz0h"
server_key: "6bc0b2dc44f1709919dac97067163cea"
server_key_iv: "8432effbb6d52148a5b792970762b9f3"
server_key_version: "encryptionkey@0"
I20260927 15:00:54.044365 19088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "978b92b3c92747f28171de4cc729c29c"
format_stamp: "Formatted at 2026-09-27 15:00:54 on dist-test-slave-pz0h"
server_key: "6bc0b2dc44f1709919dac97067163cea"
server_key_iv: "8432effbb6d52148a5b792970762b9f3"
server_key_version: "encryptionkey@0"
I20260927 15:00:54.048332 19088 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:00:54.051041 19102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:54.052377 19088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:00:54.052500 19088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "978b92b3c92747f28171de4cc729c29c"
format_stamp: "Formatted at 2026-09-27 15:00:54 on dist-test-slave-pz0h"
server_key: "6bc0b2dc44f1709919dac97067163cea"
server_key_iv: "8432effbb6d52148a5b792970762b9f3"
server_key_version: "encryptionkey@0"
I20260927 15:00:54.052627 19088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:54.095817 19088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:54.096561 19088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:54.096736 19088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:54.097334 19088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:00:54.098354 19088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:00:54.098403 19088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:54.098459 19088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:00:54.098483 19088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:54.107015 19088 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44103
I20260927 15:00:54.107046 19215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44103 every 8 connection(s)
I20260927 15:00:54.107934 19088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:00:54.108517 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 19088
I20260927 15:00:54.108631 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:00:54.108883 392 external_mini_cluster.cc:1428] Setting key 41ea98f66edb5ab333f0e35a4d3c16c0
I20260927 15:00:54.111030 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33743
--builtin_ntp_servers=127.0.98.20:40801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:00:54.117758 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:54.109678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53585 (local address 127.0.98.62:33743)
0927 15:00:54.109792 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:54.109795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:54.111297 (+ 1502us) server_negotiation.cc:407] Connection header received
0927 15:00:54.113001 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:54.113004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:54.113071 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:54.113141 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:54.114105 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.114635 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.115844 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.116060 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.116149 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:54.117262 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:54.117266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:54.117268 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:00:54.117282 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:54.117301 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:00:54.117309 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:54.117368 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:54.117514 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 15:00:54.117589 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:00:54.118649 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:54.110266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33743 (local address 127.0.98.1:53585)
0927 15:00:54.111110 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:54.111191 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:54.112773 (+ 1582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:54.113346 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:54.113354 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:54.113446 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:00:54.113961 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:54.113974 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.114769 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.114772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:54.115715 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:54.115727 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.116219 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.116224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:54.116382 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:54.116982 (+ 600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:00:54.116983 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:00:54.117025 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:00:54.117098 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:00:54.117379 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:00:54.117388 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:00:54.117394 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:00:54.117506 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 15:00:54.117824 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":137,"threads_started":1}
I20260927 15:00:54.119979 19216 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33743
I20260927 15:00:54.120218 19216 heartbeater.cc:464] Registering TS with master...
I20260927 15:00:54.120777 19216 heartbeater.cc:511] Master 127.0.98.62:33743 requested a full tablet report, sending...
I20260927 15:00:54.122437 19028 ts_manager.cc:194] Registered new tserver with Master: 978b92b3c92747f28171de4cc729c29c (127.0.98.1:44103)
I20260927 15:00:54.124120 19028 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:53585
W20260927 15:00:54.225165 19219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:54.225479 19219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:54.225520 19219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:54.229672 19219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:54.229756 19219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:54.229815 19219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:00:54.235041 19219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:54.236114 19219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:54.237488 19219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:00:54.247433 19225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:54.247506 19227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:54.247460 19219 server_base.cc:1034] running on GCE node
W20260927 15:00:54.247423 19224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:54.248061 19219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:54.248889 19219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:54.250058 19219 hybrid_clock.cc:648] HybridClock initialized: now 1790521254250033 us; error 48 us; skew 500 ppm
I20260927 15:00:54.252144 19219 webserver.cc:460] Webserver started at http://127.0.98.2:39267/ using document root <none> and password file <none>
I20260927 15:00:54.252740 19219 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:54.252813 19219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:54.253017 19219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:00:54.255281 19219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-1/data/instance:
uuid: "1eb36db6eeb941b4abb5c97792f7d3da"
format_stamp: "Formatted at 2026-09-27 15:00:54 on dist-test-slave-pz0h"
server_key: "e5bcd51643e95794918e9c29bfbfd61d"
server_key_iv: "c845c13c015ee36b7ff3cac13db08107"
server_key_version: "encryptionkey@0"
I20260927 15:00:54.255924 19219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1eb36db6eeb941b4abb5c97792f7d3da"
format_stamp: "Formatted at 2026-09-27 15:00:54 on dist-test-slave-pz0h"
server_key: "e5bcd51643e95794918e9c29bfbfd61d"
server_key_iv: "c845c13c015ee36b7ff3cac13db08107"
server_key_version: "encryptionkey@0"
I20260927 15:00:54.259845 19219 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:00:54.262463 19233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:54.263813 19219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:00:54.263949 19219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "1eb36db6eeb941b4abb5c97792f7d3da"
format_stamp: "Formatted at 2026-09-27 15:00:54 on dist-test-slave-pz0h"
server_key: "e5bcd51643e95794918e9c29bfbfd61d"
server_key_iv: "c845c13c015ee36b7ff3cac13db08107"
server_key_version: "encryptionkey@0"
I20260927 15:00:54.264070 19219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:54.293527 19219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:54.294288 19219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:54.294513 19219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:54.295166 19219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:00:54.296180 19219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:00:54.296231 19219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:54.296278 19219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:00:54.296306 19219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:54.304914 19219 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41713
I20260927 15:00:54.304941 19346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41713 every 8 connection(s)
I20260927 15:00:54.305833 19219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:00:54.307904 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 19219
I20260927 15:00:54.308023 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:00:54.308343 392 external_mini_cluster.cc:1428] Setting key cf96ff3c69c37dbebba4b6039595fc37
I20260927 15:00:54.310467 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33743
--builtin_ntp_servers=127.0.98.20:40801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:00:54.315690 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:54.307731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58863 (local address 127.0.98.62:33743)
0927 15:00:54.307855 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:54.307859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:54.309176 (+ 1317us) server_negotiation.cc:407] Connection header received
0927 15:00:54.310747 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:54.310750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:54.310830 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:54.310914 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:54.311842 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.312372 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.313508 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.313809 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.313913 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:54.315173 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:54.315178 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:54.315181 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:00:54.315198 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:54.315224 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:00:54.315235 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:54.315316 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:54.315440 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 15:00:54.315518 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:00:54.316538 19349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:54.308229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33743 (local address 127.0.98.2:58863)
0927 15:00:54.309024 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:54.309068 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:54.310502 (+ 1434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:54.311086 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:54.311094 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:54.311187 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:00:54.311694 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:54.311706 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.312502 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.312505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:54.313380 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:54.313391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.313987 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.313990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:54.314164 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:54.314844 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:00:54.314847 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:00:54.314915 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:00:54.314995 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:00:54.315313 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:00:54.315318 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:00:54.315321 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 15:00:54.315416 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 15:00:54.315680 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":165,"threads_started":1}
I20260927 15:00:54.317790 19347 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33743
I20260927 15:00:54.318043 19347 heartbeater.cc:464] Registering TS with master...
I20260927 15:00:54.318579 19347 heartbeater.cc:511] Master 127.0.98.62:33743 requested a full tablet report, sending...
I20260927 15:00:54.319713 19028 ts_manager.cc:194] Registered new tserver with Master: 1eb36db6eeb941b4abb5c97792f7d3da (127.0.98.2:41713)
I20260927 15:00:54.320282 19028 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:58863
W20260927 15:00:54.428687 19350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:54.429008 19350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:54.429064 19350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:54.433527 19350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:54.433601 19350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:54.433682 19350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:00:54.438871 19350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:54.440027 19350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:54.441449 19350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:00:54.452168 19358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:54.452168 19356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:54.452194 19355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:54.452370 19350 server_base.cc:1034] running on GCE node
I20260927 15:00:54.452947 19350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:54.453826 19350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:54.455010 19350 hybrid_clock.cc:648] HybridClock initialized: now 1790521254454998 us; error 51 us; skew 500 ppm
I20260927 15:00:54.457006 19350 webserver.cc:460] Webserver started at http://127.0.98.3:34273/ using document root <none> and password file <none>
I20260927 15:00:54.457636 19350 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:54.457706 19350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:54.457896 19350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:00:54.460170 19350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-2/data/instance:
uuid: "66055c127f56401f92759c635c447316"
format_stamp: "Formatted at 2026-09-27 15:00:54 on dist-test-slave-pz0h"
server_key: "9c92103fb3d277b7d7c66f9309c410d1"
server_key_iv: "1e3ecb4e586c89c886dfd887aadd1e64"
server_key_version: "encryptionkey@0"
I20260927 15:00:54.460779 19350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "66055c127f56401f92759c635c447316"
format_stamp: "Formatted at 2026-09-27 15:00:54 on dist-test-slave-pz0h"
server_key: "9c92103fb3d277b7d7c66f9309c410d1"
server_key_iv: "1e3ecb4e586c89c886dfd887aadd1e64"
server_key_version: "encryptionkey@0"
I20260927 15:00:54.464895 19350 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:00:54.467630 19364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:54.469148 19350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:00:54.469270 19350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "66055c127f56401f92759c635c447316"
format_stamp: "Formatted at 2026-09-27 15:00:54 on dist-test-slave-pz0h"
server_key: "9c92103fb3d277b7d7c66f9309c410d1"
server_key_iv: "1e3ecb4e586c89c886dfd887aadd1e64"
server_key_version: "encryptionkey@0"
I20260927 15:00:54.469406 19350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:54.495307 19350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:54.496044 19350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:54.496254 19350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:54.496901 19350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:00:54.497912 19350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:00:54.497963 19350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:54.498018 19350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:00:54.498045 19350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:54.508143 19350 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44191
I20260927 15:00:54.508165 19477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44191 every 8 connection(s)
I20260927 15:00:54.509202 19350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:00:54.518378 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:54.511021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40399 (local address 127.0.98.62:33743)
0927 15:00:54.511163 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:54.511168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:54.512300 (+ 1132us) server_negotiation.cc:407] Connection header received
0927 15:00:54.513639 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:54.513642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:54.513689 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:54.513750 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:54.514674 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.515245 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.516439 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.516691 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.516788 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:54.517929 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:54.517933 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:54.517935 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:00:54.517950 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:54.517971 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:00:54.517978 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:54.518042 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:54.518158 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 15:00:54.518222 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:00:54.518567 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 19350
I20260927 15:00:54.518671 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-2/wal/instance
I20260927 15:00:54.518975 392 external_mini_cluster.cc:1428] Setting key b6b83a1599f85d9dfdec45b923ee3afb
I20260927 15:00:54.519537 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:54.511488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33743 (local address 127.0.98.3:40399)
0927 15:00:54.512152 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:54.512194 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:54.513435 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:54.513917 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:54.513925 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:54.514018 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:00:54.514542 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:54.514557 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.515344 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.515347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:54.516282 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:54.516294 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.516842 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.516845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:54.517004 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:54.517654 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:00:54.517656 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:00:54.517702 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:00:54.517777 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:00:54.518046 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:00:54.518051 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:00:54.518055 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 15:00:54.518152 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 15:00:54.518447 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":121,"threads_started":1}
I20260927 15:00:54.520802 19478 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33743
I20260927 15:00:54.521064 19478 heartbeater.cc:464] Registering TS with master...
I20260927 15:00:54.521606 19478 heartbeater.cc:511] Master 127.0.98.62:33743 requested a full tablet report, sending...
I20260927 15:00:54.522640 19024 ts_manager.cc:194] Registered new tserver with Master: 66055c127f56401f92759c635c447316 (127.0.98.3:44191)
I20260927 15:00:54.523291 19024 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:40399
I20260927 15:00:54.533171 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:00:54.538509 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:54.535142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60368 (local address 127.0.98.62:33743)
0927 15:00:54.535260 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:54.535264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:54.535381 (+ 117us) server_negotiation.cc:407] Connection header received
0927 15:00:54.535450 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:54.535453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:54.535515 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:54.535599 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:54.535832 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.536353 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.537115 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.537322 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.537392 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:54.538132 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:54.538135 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:54.538137 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:00:54.538149 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:54.538167 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:00:54.538173 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:54.538253 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:54.538353 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 15:00:54.538386 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 15:00:54.541684 19024 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:00:54.544803 19024 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:00:54.559064 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:54.553179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44191 (local address 127.0.0.1:59398)
0927 15:00:54.554000 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:54.554056 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:54.554187 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:54.554680 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:54.554686 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:54.554707 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:54.554890 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:54.554899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.556052 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.556055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:54.557046 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:54.557056 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.558376 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.558380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:54.558521 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:54.558579 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 15:00:54.558715 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 15:00:54.558882 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":1,"thread_start_us":68,"threads_started":1}
I20260927 15:00:54.559103 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:54.553306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59398 (local address 127.0.98.3:44191)
0927 15:00:54.553650 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:54.553663 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:54.554311 (+ 648us) server_negotiation.cc:407] Connection header received
0927 15:00:54.554381 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:54.554384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:54.554542 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:54.554669 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:54.555033 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.555920 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.557171 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.558002 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.558107 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:54.558313 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:54.558765 (+ 452us) server_negotiation.cc:299] Negotiation successful
0927 15:00:54.558912 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":110,"threads_started":1}
I20260927 15:00:54.559916 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:54.553717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44103 (local address 127.0.0.1:38398)
0927 15:00:54.554120 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:54.554133 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:54.554221 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:54.555592 (+ 1371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:54.555595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:54.555613 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:54.555727 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:54.555733 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.557324 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.557327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:54.558427 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:54.558436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.559535 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.559539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:54.559628 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:54.559649 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:00:54.559708 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 15:00:54.559777 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":1,"thread_start_us":102,"threads_started":1}
I20260927 15:00:54.560014 19494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:54.554545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38398 (local address 127.0.98.1:44103)
0927 15:00:54.554844 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:54.554850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:54.554873 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:00:54.555107 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:54.555113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:54.555300 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:54.555878 (+ 578us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:54.556016 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.557210 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.558573 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.559423 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.559525 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:54.559726 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:54.559777 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 15:00:54.559871 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":91,"threads_started":1}
I20260927 15:00:54.560240 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:54.553916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41713 (local address 127.0.0.1:49950)
0927 15:00:54.554930 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:54.554942 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:54.555028 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:54.555936 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:54.555939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:54.555958 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:54.556040 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:54.556046 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.557390 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.557394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:54.558559 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:54.558569 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.559861 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.559864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:54.559949 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:54.559969 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 15:00:54.560034 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 15:00:54.560100 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":957,"mutex_wait_us":29,"thread_start_us":89,"threads_started":1}
I20260927 15:00:54.560668 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:54.554146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49950 (local address 127.0.98.2:41713)
0927 15:00:54.554473 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:54.554483 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:54.555494 (+ 1011us) server_negotiation.cc:407] Connection header received
0927 15:00:54.555580 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:54.555585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:54.555765 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:54.555941 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:54.556123 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.557307 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.558681 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.559524 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.559626 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:54.559800 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:54.560396 (+ 596us) server_negotiation.cc:299] Negotiation successful
0927 15:00:54.560512 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":138,"threads_started":1}
I20260927 15:00:54.562616 19150 tablet_service.cc:1467] Processing CreateTablet for tablet 85c155918d4248479911d33736524fe5 (DEFAULT_TABLE table=test-table [id=c4808a61efe749aaa49d9f86fa910cee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:00:54.562623 19412 tablet_service.cc:1467] Processing CreateTablet for tablet 85c155918d4248479911d33736524fe5 (DEFAULT_TABLE table=test-table [id=c4808a61efe749aaa49d9f86fa910cee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:00:54.563625 19150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85c155918d4248479911d33736524fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:00:54.563235 19281 tablet_service.cc:1467] Processing CreateTablet for tablet 85c155918d4248479911d33736524fe5 (DEFAULT_TABLE table=test-table [id=c4808a61efe749aaa49d9f86fa910cee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:00:54.563687 19412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85c155918d4248479911d33736524fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:00:54.564210 19281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85c155918d4248479911d33736524fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:00:54.571554 19497 tablet_bootstrap.cc:492] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da: Bootstrap starting.
I20260927 15:00:54.571938 19496 tablet_bootstrap.cc:492] T 85c155918d4248479911d33736524fe5 P 66055c127f56401f92759c635c447316: Bootstrap starting.
I20260927 15:00:54.571979 19495 tablet_bootstrap.cc:492] T 85c155918d4248479911d33736524fe5 P 978b92b3c92747f28171de4cc729c29c: Bootstrap starting.
I20260927 15:00:54.573983 19496 tablet_bootstrap.cc:654] T 85c155918d4248479911d33736524fe5 P 66055c127f56401f92759c635c447316: Neither blocks nor log segments found. Creating new log.
I20260927 15:00:54.573999 19497 tablet_bootstrap.cc:654] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da: Neither blocks nor log segments found. Creating new log.
I20260927 15:00:54.573999 19495 tablet_bootstrap.cc:654] T 85c155918d4248479911d33736524fe5 P 978b92b3c92747f28171de4cc729c29c: Neither blocks nor log segments found. Creating new log.
I20260927 15:00:54.574827 19496 log.cc:824] T 85c155918d4248479911d33736524fe5 P 66055c127f56401f92759c635c447316: Log is configured to *not* fsync() on all Append() calls
I20260927 15:00:54.574851 19495 log.cc:824] T 85c155918d4248479911d33736524fe5 P 978b92b3c92747f28171de4cc729c29c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:00:54.574851 19497 log.cc:824] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da: Log is configured to *not* fsync() on all Append() calls
I20260927 15:00:54.576839 19497 tablet_bootstrap.cc:492] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da: No bootstrap required, opened a new log
I20260927 15:00:54.576849 19496 tablet_bootstrap.cc:492] T 85c155918d4248479911d33736524fe5 P 66055c127f56401f92759c635c447316: No bootstrap required, opened a new log
I20260927 15:00:54.577061 19496 ts_tablet_manager.cc:1397] T 85c155918d4248479911d33736524fe5 P 66055c127f56401f92759c635c447316: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 15:00:54.577061 19497 ts_tablet_manager.cc:1397] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:00:54.577261 19495 tablet_bootstrap.cc:492] T 85c155918d4248479911d33736524fe5 P 978b92b3c92747f28171de4cc729c29c: No bootstrap required, opened a new log
I20260927 15:00:54.577494 19495 ts_tablet_manager.cc:1397] T 85c155918d4248479911d33736524fe5 P 978b92b3c92747f28171de4cc729c29c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:00:54.580781 19497 raft_consensus.cc:348] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66055c127f56401f92759c635c447316" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44191 } } peers { permanent_uuid: "978b92b3c92747f28171de4cc729c29c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44103 } } peers { permanent_uuid: "1eb36db6eeb941b4abb5c97792f7d3da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41713 } }
I20260927 15:00:54.581063 19497 raft_consensus.cc:374] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:00:54.581115 19497 raft_consensus.cc:729] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb36db6eeb941b4abb5c97792f7d3da, State: Initialized, Role: FOLLOWER
I20260927 15:00:54.581627 19497 consensus_queue.cc:260] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66055c127f56401f92759c635c447316" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44191 } } peers { permanent_uuid: "978b92b3c92747f28171de4cc729c29c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44103 } } peers { permanent_uuid: "1eb36db6eeb941b4abb5c97792f7d3da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41713 } }
I20260927 15:00:54.581559 19496 raft_consensus.cc:348] T 85c155918d4248479911d33736524fe5 P 66055c127f56401f92759c635c447316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66055c127f56401f92759c635c447316" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44191 } } peers { permanent_uuid: "978b92b3c92747f28171de4cc729c29c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44103 } } peers { permanent_uuid: "1eb36db6eeb941b4abb5c97792f7d3da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41713 } }
I20260927 15:00:54.581559 19495 raft_consensus.cc:348] T 85c155918d4248479911d33736524fe5 P 978b92b3c92747f28171de4cc729c29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66055c127f56401f92759c635c447316" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44191 } } peers { permanent_uuid: "978b92b3c92747f28171de4cc729c29c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44103 } } peers { permanent_uuid: "1eb36db6eeb941b4abb5c97792f7d3da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41713 } }
I20260927 15:00:54.581835 19495 raft_consensus.cc:374] T 85c155918d4248479911d33736524fe5 P 978b92b3c92747f28171de4cc729c29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:00:54.581835 19496 raft_consensus.cc:374] T 85c155918d4248479911d33736524fe5 P 66055c127f56401f92759c635c447316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:00:54.581887 19495 raft_consensus.cc:729] T 85c155918d4248479911d33736524fe5 P 978b92b3c92747f28171de4cc729c29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 978b92b3c92747f28171de4cc729c29c, State: Initialized, Role: FOLLOWER
I20260927 15:00:54.581892 19496 raft_consensus.cc:729] T 85c155918d4248479911d33736524fe5 P 66055c127f56401f92759c635c447316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66055c127f56401f92759c635c447316, State: Initialized, Role: FOLLOWER
I20260927 15:00:54.582374 19347 heartbeater.cc:503] Master 127.0.98.62:33743 was elected leader, sending a full tablet report...
I20260927 15:00:54.582381 19496 consensus_queue.cc:260] T 85c155918d4248479911d33736524fe5 P 66055c127f56401f92759c635c447316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66055c127f56401f92759c635c447316" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44191 } } peers { permanent_uuid: "978b92b3c92747f28171de4cc729c29c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44103 } } peers { permanent_uuid: "1eb36db6eeb941b4abb5c97792f7d3da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41713 } }
I20260927 15:00:54.582365 19495 consensus_queue.cc:260] T 85c155918d4248479911d33736524fe5 P 978b92b3c92747f28171de4cc729c29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66055c127f56401f92759c635c447316" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44191 } } peers { permanent_uuid: "978b92b3c92747f28171de4cc729c29c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44103 } } peers { permanent_uuid: "1eb36db6eeb941b4abb5c97792f7d3da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41713 } }
I20260927 15:00:54.583062 19497 ts_tablet_manager.cc:1428] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 15:00:54.583395 19496 ts_tablet_manager.cc:1428] T 85c155918d4248479911d33736524fe5 P 66055c127f56401f92759c635c447316: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 15:00:54.583395 19495 ts_tablet_manager.cc:1428] T 85c155918d4248479911d33736524fe5 P 978b92b3c92747f28171de4cc729c29c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:00:54.583417 19502 raft_consensus.cc:482] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:00:54.583564 19502 raft_consensus.cc:504] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66055c127f56401f92759c635c447316" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44191 } } peers { permanent_uuid: "978b92b3c92747f28171de4cc729c29c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44103 } } peers { permanent_uuid: "1eb36db6eeb941b4abb5c97792f7d3da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41713 } }
I20260927 15:00:54.583766 19478 heartbeater.cc:503] Master 127.0.98.62:33743 was elected leader, sending a full tablet report...
I20260927 15:00:54.584157 19216 heartbeater.cc:503] Master 127.0.98.62:33743 was elected leader, sending a full tablet report...
I20260927 15:00:54.587368 19502 leader_election.cc:288] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66055c127f56401f92759c635c447316 (127.0.98.3:44191), 978b92b3c92747f28171de4cc729c29c (127.0.98.1:44103)
I20260927 15:00:54.589685 19349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:54.585014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44191 (local address 127.0.98.2:60455)
0927 15:00:54.585463 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:54.585480 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:54.585577 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:54.586031 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:54.586034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:54.586055 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:54.586175 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:54.586183 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.587192 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.587196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:54.588330 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:54.588341 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.589251 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.589256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:54.589371 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:54.589396 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 15:00:54.589460 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 15:00:54.589553 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356}
I20260927 15:00:54.590040 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:54.585310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60455 (local address 127.0.98.3:44191)
0927 15:00:54.585732 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:54.585736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:54.585750 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:00:54.585799 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:54.585802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:54.585845 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:54.585912 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:54.586314 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.587056 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.588458 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.589022 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.589124 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:54.589185 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:54.589846 (+ 661us) server_negotiation.cc:299] Negotiation successful
0927 15:00:54.589922 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343}
I20260927 15:00:54.590564 19432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85c155918d4248479911d33736524fe5" candidate_uuid: "1eb36db6eeb941b4abb5c97792f7d3da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66055c127f56401f92759c635c447316" is_pre_election: true
I20260927 15:00:54.590873 19432 raft_consensus.cc:2454] T 85c155918d4248479911d33736524fe5 P 66055c127f56401f92759c635c447316 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eb36db6eeb941b4abb5c97792f7d3da in term 0.
I20260927 15:00:54.591300 19234 leader_election.cc:302] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb36db6eeb941b4abb5c97792f7d3da, 66055c127f56401f92759c635c447316; no voters:
I20260927 15:00:54.591534 19502 raft_consensus.cc:2781] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:00:54.591619 19502 raft_consensus.cc:482] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:00:54.591658 19502 raft_consensus.cc:3037] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:00:54.592643 19502 raft_consensus.cc:504] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66055c127f56401f92759c635c447316" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44191 } } peers { permanent_uuid: "978b92b3c92747f28171de4cc729c29c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44103 } } peers { permanent_uuid: "1eb36db6eeb941b4abb5c97792f7d3da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41713 } }
I20260927 15:00:54.592995 19502 leader_election.cc:288] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da [CANDIDATE]: Term 1 election: Requested vote from peers 66055c127f56401f92759c635c447316 (127.0.98.3:44191), 978b92b3c92747f28171de4cc729c29c (127.0.98.1:44103)
I20260927 15:00:54.593155 19494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:54.587785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50929 (local address 127.0.98.1:44103)
0927 15:00:54.587893 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:54.587897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:54.590016 (+ 2119us) server_negotiation.cc:407] Connection header received
0927 15:00:54.590153 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:54.590157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:54.590217 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:54.590288 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:54.590521 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.591131 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.592219 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.592599 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.592702 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:54.592760 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:54.592952 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 15:00:54.593039 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 15:00:54.593173 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:54.587743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44103 (local address 127.0.98.2:50929)
0927 15:00:54.589919 (+ 2176us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:54.589934 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:54.590058 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:54.590304 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:54.590307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:54.590324 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:54.590426 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:54.590430 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.591235 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.591239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:54.592101 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:54.592110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.592735 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.592739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:54.592824 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:54.592846 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:00:54.592917 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 15:00:54.592986 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1196,"thread_start_us":114,"threads_started":1}
I20260927 15:00:54.593446 19432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85c155918d4248479911d33736524fe5" candidate_uuid: "1eb36db6eeb941b4abb5c97792f7d3da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66055c127f56401f92759c635c447316"
I20260927 15:00:54.593575 19432 raft_consensus.cc:3037] T 85c155918d4248479911d33736524fe5 P 66055c127f56401f92759c635c447316 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:00:54.594588 19432 raft_consensus.cc:2454] T 85c155918d4248479911d33736524fe5 P 66055c127f56401f92759c635c447316 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eb36db6eeb941b4abb5c97792f7d3da in term 1.
I20260927 15:00:54.594939 19234 leader_election.cc:302] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb36db6eeb941b4abb5c97792f7d3da, 66055c127f56401f92759c635c447316; no voters:
I20260927 15:00:54.595112 19502 raft_consensus.cc:2781] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:00:54.595204 19502 raft_consensus.cc:686] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da [term 1 LEADER]: Becoming Leader. State: Replica: 1eb36db6eeb941b4abb5c97792f7d3da, State: Running, Role: LEADER
I20260927 15:00:54.595391 19169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85c155918d4248479911d33736524fe5" candidate_uuid: "1eb36db6eeb941b4abb5c97792f7d3da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "978b92b3c92747f28171de4cc729c29c"
I20260927 15:00:54.595366 19170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85c155918d4248479911d33736524fe5" candidate_uuid: "1eb36db6eeb941b4abb5c97792f7d3da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "978b92b3c92747f28171de4cc729c29c" is_pre_election: true
I20260927 15:00:54.595607 19169 raft_consensus.cc:3037] T 85c155918d4248479911d33736524fe5 P 978b92b3c92747f28171de4cc729c29c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:00:54.596675 19169 raft_consensus.cc:2454] T 85c155918d4248479911d33736524fe5 P 978b92b3c92747f28171de4cc729c29c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eb36db6eeb941b4abb5c97792f7d3da in term 1.
I20260927 15:00:54.597223 19502 consensus_queue.cc:237] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66055c127f56401f92759c635c447316" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44191 } } peers { permanent_uuid: "978b92b3c92747f28171de4cc729c29c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44103 } } peers { permanent_uuid: "1eb36db6eeb941b4abb5c97792f7d3da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41713 } }
I20260927 15:00:54.600859 19024 catalog_manager.cc:5495] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eb36db6eeb941b4abb5c97792f7d3da (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1eb36db6eeb941b4abb5c97792f7d3da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66055c127f56401f92759c635c447316" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "978b92b3c92747f28171de4cc729c29c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eb36db6eeb941b4abb5c97792f7d3da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41713 } health_report { overall_health: HEALTHY } } }
W20260927 15:00:54.609282 19217 tablet.cc:2369] T 85c155918d4248479911d33736524fe5 P 978b92b3c92747f28171de4cc729c29c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:00:54.612597 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:54.609046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49966 (local address 127.0.98.2:41713)
0927 15:00:54.609183 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:54.609187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:54.609204 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:00:54.609277 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:54.609279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:54.609335 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:54.609443 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:00:54.609721 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.610230 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.611119 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:54.611337 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:54.611427 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:54.611750 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:00:54.612177 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:00:54.612306 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:54.612420 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 15:00:54.612460 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:00:54.618669 19169 raft_consensus.cc:1261] T 85c155918d4248479911d33736524fe5 P 978b92b3c92747f28171de4cc729c29c [term 1 FOLLOWER]: Refusing update from remote peer 1eb36db6eeb941b4abb5c97792f7d3da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:00:54.618731 19432 raft_consensus.cc:1261] T 85c155918d4248479911d33736524fe5 P 66055c127f56401f92759c635c447316 [term 1 FOLLOWER]: Refusing update from remote peer 1eb36db6eeb941b4abb5c97792f7d3da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:00:54.619442 19501 consensus_queue.cc:1035] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da [LEADER]: Connected to new peer: Peer: permanent_uuid: "66055c127f56401f92759c635c447316" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:00:54.619647 19502 consensus_queue.cc:1035] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da [LEADER]: Connected to new peer: Peer: permanent_uuid: "978b92b3c92747f28171de4cc729c29c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:00:54.628253 19510 mvcc.cc:205] Tried to move back new op lower bound from 7333975058910957568 to 7333975058833035264. Current Snapshot: MvccSnapshot[applied={T|T < 7333975058910957568}]
I20260927 15:00:54.628456 19508 mvcc.cc:205] Tried to move back new op lower bound from 7333975058910957568 to 7333975058833035264. Current Snapshot: MvccSnapshot[applied={T|T < 7333975058910957568}]
I20260927 15:00:54.630288 19509 mvcc.cc:205] Tried to move back new op lower bound from 7333975058910957568 to 7333975058833035264. Current Snapshot: MvccSnapshot[applied={T|T < 7333975058910957568}]
I20260927 15:00:54.637976 19027 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:00:54.638159 19027 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:00:54.640894 19027 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bdedab0848e6442da46b2617381213fb: Sending DeleteTablet for 3 replicas of tablet 85c155918d4248479911d33736524fe5
I20260927 15:00:54.641780 19412 tablet_service.cc:1514] Processing DeleteTablet for tablet 85c155918d4248479911d33736524fe5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:00:54 UTC) from {username='slave'} at 127.0.0.1:59398
I20260927 15:00:54.641781 19281 tablet_service.cc:1514] Processing DeleteTablet for tablet 85c155918d4248479911d33736524fe5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:00:54 UTC) from {username='slave'} at 127.0.0.1:49950
I20260927 15:00:54.641821 19150 tablet_service.cc:1514] Processing DeleteTablet for tablet 85c155918d4248479911d33736524fe5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:00:54 UTC) from {username='slave'} at 127.0.0.1:38398
I20260927 15:00:54.642342 19517 tablet_replica.cc:330] T 85c155918d4248479911d33736524fe5 P 978b92b3c92747f28171de4cc729c29c: stopping tablet replica
I20260927 15:00:54.642546 19518 tablet_replica.cc:330] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da: stopping tablet replica
I20260927 15:00:54.642884 19518 raft_consensus.cc:2229] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:00:54.642885 19517 raft_consensus.cc:2229] T 85c155918d4248479911d33736524fe5 P 978b92b3c92747f28171de4cc729c29c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:00:54.643325 19518 raft_consensus.cc:2258] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:00:54.643383 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 19088
I20260927 15:00:54.643352 19517 raft_consensus.cc:2258] T 85c155918d4248479911d33736524fe5 P 978b92b3c92747f28171de4cc729c29c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:00:54.644765 19519 tablet_replica.cc:330] T 85c155918d4248479911d33736524fe5 P 66055c127f56401f92759c635c447316: stopping tablet replica
I20260927 15:00:54.645098 19519 raft_consensus.cc:2229] T 85c155918d4248479911d33736524fe5 P 66055c127f56401f92759c635c447316 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:00:54.645437 19519 raft_consensus.cc:2258] T 85c155918d4248479911d33736524fe5 P 66055c127f56401f92759c635c447316 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:00:54.646510 19518 ts_tablet_manager.cc:1905] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:00:54.647238 19519 ts_tablet_manager.cc:1905] T 85c155918d4248479911d33736524fe5 P 66055c127f56401f92759c635c447316: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:00:54.650192 19519 ts_tablet_manager.cc:1918] T 85c155918d4248479911d33736524fe5 P 66055c127f56401f92759c635c447316: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:00:54.650318 19519 log.cc:1196] T 85c155918d4248479911d33736524fe5 P 66055c127f56401f92759c635c447316: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-2/wal/wals/85c155918d4248479911d33736524fe5
I20260927 15:00:54.650262 19518 ts_tablet_manager.cc:1918] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:00:54.650599 19518 log.cc:1196] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790521004237705-392-0/minicluster-data/ts-1/wal/wals/85c155918d4248479911d33736524fe5
I20260927 15:00:54.650607 19519 ts_tablet_manager.cc:1939] T 85c155918d4248479911d33736524fe5 P 66055c127f56401f92759c635c447316: Deleting consensus metadata
I20260927 15:00:54.650991 19518 ts_tablet_manager.cc:1939] T 85c155918d4248479911d33736524fe5 P 1eb36db6eeb941b4abb5c97792f7d3da: Deleting consensus metadata
W20260927 15:00:54.652027 19013 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44103 (error 108)
I20260927 15:00:54.652201 19015 catalog_manager.cc:4841] TS 1eb36db6eeb941b4abb5c97792f7d3da (127.0.98.2:41713): tablet 85c155918d4248479911d33736524fe5 (table test-table [id=c4808a61efe749aaa49d9f86fa910cee]) successfully deleted
I20260927 15:00:54.652381 19012 catalog_manager.cc:4841] TS 66055c127f56401f92759c635c447316 (127.0.98.3:44191): tablet 85c155918d4248479911d33736524fe5 (table test-table [id=c4808a61efe749aaa49d9f86fa910cee]) successfully deleted
I20260927 15:00:54.652902 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 19219
I20260927 15:00:54.653738 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:54.653091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44103 (local address 127.0.0.1:38404)
0927 15:00:54.653565 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:54.653641 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386}
W20260927 15:00:54.654256 19013 catalog_manager.cc:4584] TS 978b92b3c92747f28171de4cc729c29c (127.0.98.1:44103): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 85c155918d4248479911d33736524fe5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44103: connect: Connection refused (error 111)
I20260927 15:00:54.658229 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 19350
I20260927 15:00:54.666004 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 18997
2026-09-27T15:00:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1055 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:00:54 dist-test-slave-pz0h krb5kdc[19523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:00:54 dist-test-slave-pz0h krb5kdc[19523](info): set up 2 sockets
Sep 27 15:00:54 dist-test-slave-pz0h krb5kdc[19523](info): commencing operation
krb5kdc: starting...
W20260927 15:00:55.614547 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 15:00:55 dist-test-slave-pz0h krb5kdc[19523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:00:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:00:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:00:55.736191 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45941
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:00:55.852779 19536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:55.853112 19536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:55.853154 19536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:55.859017 19536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:00:55.859122 19536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:55.859150 19536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:55.859174 19536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:00:55.859194 19536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:00:55.865465 19536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:55.866598 19536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:55.867676 19536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:00:55.877249 19541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:55.877211 19542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:55.877205 19544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:55.877696 19536 server_base.cc:1034] running on GCE node
I20260927 15:00:55.878302 19536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:55.879819 19536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:55.881031 19536 hybrid_clock.cc:648] HybridClock initialized: now 1790521255881015 us; error 42 us; skew 500 ppm
Sep 27 15:00:55 dist-test-slave-pz0h krb5kdc[19523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:00:55.884249 19536 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:00:55.885336 19536 webserver.cc:460] Webserver started at http://127.0.98.62:46431/ using document root <none> and password file <none>
I20260927 15:00:55.885978 19536 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:55.886060 19536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:55.886283 19536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:00:55.888636 19536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "f601f7a0557943638ca67afbd2fa6490"
format_stamp: "Formatted at 2026-09-27 15:00:55 on dist-test-slave-pz0h"
server_key: "305b76bafed9fea658110ab12d2add31"
server_key_iv: "4fee927062f9fb0a1c2eabc9f2b34f8b"
server_key_version: "encryptionkey@0"
I20260927 15:00:55.889242 19536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "f601f7a0557943638ca67afbd2fa6490"
format_stamp: "Formatted at 2026-09-27 15:00:55 on dist-test-slave-pz0h"
server_key: "305b76bafed9fea658110ab12d2add31"
server_key_iv: "4fee927062f9fb0a1c2eabc9f2b34f8b"
server_key_version: "encryptionkey@0"
I20260927 15:00:55.894266 19536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:00:55.897953 19551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:55.899573 19536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:00:55.899739 19536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "f601f7a0557943638ca67afbd2fa6490"
format_stamp: "Formatted at 2026-09-27 15:00:55 on dist-test-slave-pz0h"
server_key: "305b76bafed9fea658110ab12d2add31"
server_key_iv: "4fee927062f9fb0a1c2eabc9f2b34f8b"
server_key_version: "encryptionkey@0"
I20260927 15:00:55.899894 19536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:55.914968 19536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:55.918457 19536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:55.918669 19536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:55.926549 19536 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45941
I20260927 15:00:55.926597 19603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45941 every 8 connection(s)
I20260927 15:00:55.927661 19536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:00:55.930713 19604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:00:55.933523 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 19536
I20260927 15:00:55.933667 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 15:00:55.933911 392 external_mini_cluster.cc:1428] Setting key 1a715c90d4f3d48c723b209b0700f71b
I20260927 15:00:55.937714 19604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f601f7a0557943638ca67afbd2fa6490: Bootstrap starting.
I20260927 15:00:55.940586 19604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f601f7a0557943638ca67afbd2fa6490: Neither blocks nor log segments found. Creating new log.
I20260927 15:00:55.941929 19604 log.cc:824] T 00000000000000000000000000000000 P f601f7a0557943638ca67afbd2fa6490: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:00:55 dist-test-slave-pz0h krb5kdc[19523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:00:55.944800 19604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f601f7a0557943638ca67afbd2fa6490: No bootstrap required, opened a new log
I20260927 15:00:55.949062 19604 raft_consensus.cc:348] T 00000000000000000000000000000000 P f601f7a0557943638ca67afbd2fa6490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f601f7a0557943638ca67afbd2fa6490" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45941 } }
I20260927 15:00:55.949347 19604 raft_consensus.cc:374] T 00000000000000000000000000000000 P f601f7a0557943638ca67afbd2fa6490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:00:55.949373 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:55.935337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37294 (local address 127.0.98.62:45941)
0927 15:00:55.935789 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:55.935805 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:55.935843 (+ 38us) server_negotiation.cc:407] Connection header received
0927 15:00:55.936513 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:55.936531 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:55.936898 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:55.937218 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:55.938354 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:55.939480 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:55.940680 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:55.941249 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:55.941385 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:55.944600 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:55.944605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:55.944620 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:00:55.944652 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:55.946309 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:55.946911 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:55.946916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:55.946921 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:55.947047 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:55.947329 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:55.947331 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:55.947333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:55.947774 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:55.947936 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:55.948229 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 15:00:55.948541 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":127,"threads_started":1}
I20260927 15:00:55.949410 19604 raft_consensus.cc:729] T 00000000000000000000000000000000 P f601f7a0557943638ca67afbd2fa6490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f601f7a0557943638ca67afbd2fa6490, State: Initialized, Role: FOLLOWER
I20260927 15:00:55.949920 19604 consensus_queue.cc:260] T 00000000000000000000000000000000 P f601f7a0557943638ca67afbd2fa6490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f601f7a0557943638ca67afbd2fa6490" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45941 } }
I20260927 15:00:55.950098 19604 raft_consensus.cc:388] T 00000000000000000000000000000000 P f601f7a0557943638ca67afbd2fa6490 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:00:55.950160 19604 raft_consensus.cc:482] T 00000000000000000000000000000000 P f601f7a0557943638ca67afbd2fa6490 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:00:55.950254 19604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f601f7a0557943638ca67afbd2fa6490 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:00:55.951316 19604 raft_consensus.cc:504] T 00000000000000000000000000000000 P f601f7a0557943638ca67afbd2fa6490 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f601f7a0557943638ca67afbd2fa6490" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45941 } }
I20260927 15:00:55.951661 19604 leader_election.cc:302] T 00000000000000000000000000000000 P f601f7a0557943638ca67afbd2fa6490 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f601f7a0557943638ca67afbd2fa6490; no voters:
I20260927 15:00:55.951983 19604 leader_election.cc:288] T 00000000000000000000000000000000 P f601f7a0557943638ca67afbd2fa6490 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:00:55.952093 19609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f601f7a0557943638ca67afbd2fa6490 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:00:55.952299 19609 raft_consensus.cc:686] T 00000000000000000000000000000000 P f601f7a0557943638ca67afbd2fa6490 [term 1 LEADER]: Becoming Leader. State: Replica: f601f7a0557943638ca67afbd2fa6490, State: Running, Role: LEADER
I20260927 15:00:55.952641 19609 consensus_queue.cc:237] T 00000000000000000000000000000000 P f601f7a0557943638ca67afbd2fa6490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f601f7a0557943638ca67afbd2fa6490" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45941 } }
I20260927 15:00:55.953251 19604 sys_catalog.cc:564] T 00000000000000000000000000000000 P f601f7a0557943638ca67afbd2fa6490 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:00:55.954389 19610 sys_catalog.cc:455] T 00000000000000000000000000000000 P f601f7a0557943638ca67afbd2fa6490 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f601f7a0557943638ca67afbd2fa6490" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f601f7a0557943638ca67afbd2fa6490" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45941 } } }
I20260927 15:00:55.954396 19611 sys_catalog.cc:455] T 00000000000000000000000000000000 P f601f7a0557943638ca67afbd2fa6490 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f601f7a0557943638ca67afbd2fa6490. Latest consensus state: current_term: 1 leader_uuid: "f601f7a0557943638ca67afbd2fa6490" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f601f7a0557943638ca67afbd2fa6490" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45941 } } }
I20260927 15:00:55.954550 19611 sys_catalog.cc:458] T 00000000000000000000000000000000 P f601f7a0557943638ca67afbd2fa6490 [sys.catalog]: This master's current role is: LEADER
I20260927 15:00:55.954550 19610 sys_catalog.cc:458] T 00000000000000000000000000000000 P f601f7a0557943638ca67afbd2fa6490 [sys.catalog]: This master's current role is: LEADER
I20260927 15:00:55.955008 19618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:00:55.958899 19618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:00:55.965502 19618 catalog_manager.cc:1347] Generated new cluster ID: f34966c99eee4781a1a1632e0f070eef
I20260927 15:00:55.965606 19618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:00:55.977919 19618 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:00:55.979135 19618 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:00:55.988533 19618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f601f7a0557943638ca67afbd2fa6490: Generated new TSK 0
I20260927 15:00:55.989451 19618 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 15:00:56.021876 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45941
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:00:56.141119 19630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:56.141423 19630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:56.141469 19630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:56.145709 19630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:56.145793 19630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:56.145855 19630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:00:56.152426 19630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:56.153843 19630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:56.155068 19630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:00:56.165686 19638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:56.165675 19636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:56.165675 19635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:56.165982 19630 server_base.cc:1034] running on GCE node
I20260927 15:00:56.166527 19630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:56.167317 19630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:56.168511 19630 hybrid_clock.cc:648] HybridClock initialized: now 1790521256168477 us; error 38 us; skew 500 ppm
Sep 27 15:00:56 dist-test-slave-pz0h krb5kdc[19523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:00:56.171481 19630 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 15:00:56.172488 19630 webserver.cc:460] Webserver started at http://127.0.98.1:34635/ using document root <none> and password file <none>
I20260927 15:00:56.173082 19630 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:56.173152 19630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:56.173334 19630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:00:56.175729 19630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-0/data/instance:
uuid: "c14239daf5cb4158aba377093d9531f5"
format_stamp: "Formatted at 2026-09-27 15:00:56 on dist-test-slave-pz0h"
server_key: "9d5d6eb7d36566ec37f3a77e98b16c9a"
server_key_iv: "d15e2d96e655d2d5660e40929b45ffd1"
server_key_version: "encryptionkey@0"
I20260927 15:00:56.176332 19630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c14239daf5cb4158aba377093d9531f5"
format_stamp: "Formatted at 2026-09-27 15:00:56 on dist-test-slave-pz0h"
server_key: "9d5d6eb7d36566ec37f3a77e98b16c9a"
server_key_iv: "d15e2d96e655d2d5660e40929b45ffd1"
server_key_version: "encryptionkey@0"
I20260927 15:00:56.181293 19630 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:00:56.184245 19645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:56.185672 19630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:00:56.185812 19630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "c14239daf5cb4158aba377093d9531f5"
format_stamp: "Formatted at 2026-09-27 15:00:56 on dist-test-slave-pz0h"
server_key: "9d5d6eb7d36566ec37f3a77e98b16c9a"
server_key_iv: "d15e2d96e655d2d5660e40929b45ffd1"
server_key_version: "encryptionkey@0"
I20260927 15:00:56.185948 19630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:56.204771 19630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:56.208729 19630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:56.208920 19630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:56.209523 19630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:00:56.210419 19630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:00:56.210469 19630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:56.210512 19630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:00:56.210530 19630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:56.219499 19630 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46733
I20260927 15:00:56.219525 19758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46733 every 8 connection(s)
I20260927 15:00:56.220511 19630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:00:56.229080 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 19630
I20260927 15:00:56.229235 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:00:56.229532 392 external_mini_cluster.cc:1428] Setting key b777449df94f4cc61dd98d54b29b46b0
Sep 27 15:00:56 dist-test-slave-pz0h krb5kdc[19523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:00:56.237975 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:56.223262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43779 (local address 127.0.98.62:45941)
0927 15:00:56.223492 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:56.223496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:56.224706 (+ 1210us) server_negotiation.cc:407] Connection header received
0927 15:00:56.226159 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:56.226163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:56.226231 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:56.226318 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:56.228394 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:56.229091 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:56.231005 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:56.231261 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:56.231364 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:56.234161 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:56.234165 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:56.234167 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:00:56.234196 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:56.235721 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:56.236329 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:56.236333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:56.236335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:56.236429 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:56.236776 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:56.236779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:56.236782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:56.236988 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:56.237077 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:56.237650 (+ 573us) server_negotiation.cc:299] Negotiation successful
0927 15:00:56.237795 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260927 15:00:56.238960 19761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:56.223708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45941 (local address 127.0.98.1:43779)
0927 15:00:56.224444 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:56.224514 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:56.225838 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:56.226663 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:56.226676 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:56.227248 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:00:56.228142 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:56.228168 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:56.229335 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:56.229340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:56.230839 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:56.230867 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:56.231436 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:56.231441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:56.231635 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:56.232312 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:00:56.232331 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:00:56.233931 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:00:56.235874 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:00:56.235882 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:00:56.235904 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:00:56.236223 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:00:56.236524 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:00:56.236527 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:00:56.236530 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:00:56.236684 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:00:56.237098 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:00:56.237106 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:00:56.237402 (+ 296us) client_negotiation.cc:769] Sending connection context
0927 15:00:56.237592 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 15:00:56.237994 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":325,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 15:00:56.240458 19759 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45941
I20260927 15:00:56.240708 19759 heartbeater.cc:464] Registering TS with master...
I20260927 15:00:56.241214 19759 heartbeater.cc:511] Master 127.0.98.62:45941 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 15:00:56.242993 19568 ts_manager.cc:194] Registered new tserver with Master: c14239daf5cb4158aba377093d9531f5 (127.0.98.1:46733)
I20260927 15:00:56.244210 19568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 15:00:56.258280 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45941
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:00:56.376442 19764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:56.376760 19764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:56.376806 19764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:56.381140 19764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:56.381233 19764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:56.381295 19764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:00:56.387074 19764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:56.388250 19764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:56.389346 19764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:00:56.399320 19769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:56.399320 19770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:56.399320 19772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:56.399706 19764 server_base.cc:1034] running on GCE node
I20260927 15:00:56.400151 19764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:56.400812 19764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:56.402004 19764 hybrid_clock.cc:648] HybridClock initialized: now 1790521256401973 us; error 43 us; skew 500 ppm
Sep 27 15:00:56 dist-test-slave-pz0h krb5kdc[19523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:00:56.405007 19764 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 15:00:56.406008 19764 webserver.cc:460] Webserver started at http://127.0.98.2:33313/ using document root <none> and password file <none>
I20260927 15:00:56.406595 19764 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:56.406669 19764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:56.406944 19764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:00:56.409140 19764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-1/data/instance:
uuid: "44b99506d67d48ae9d54c264a1298bb5"
format_stamp: "Formatted at 2026-09-27 15:00:56 on dist-test-slave-pz0h"
server_key: "73e1f70b2b7b21dc52a3b7627b984779"
server_key_iv: "5b1f3b6effa1b206da200dc4b949375f"
server_key_version: "encryptionkey@0"
I20260927 15:00:56.409721 19764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "44b99506d67d48ae9d54c264a1298bb5"
format_stamp: "Formatted at 2026-09-27 15:00:56 on dist-test-slave-pz0h"
server_key: "73e1f70b2b7b21dc52a3b7627b984779"
server_key_iv: "5b1f3b6effa1b206da200dc4b949375f"
server_key_version: "encryptionkey@0"
I20260927 15:00:56.413801 19764 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:00:56.416447 19779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:56.417747 19764 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:00:56.417896 19764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "44b99506d67d48ae9d54c264a1298bb5"
format_stamp: "Formatted at 2026-09-27 15:00:56 on dist-test-slave-pz0h"
server_key: "73e1f70b2b7b21dc52a3b7627b984779"
server_key_iv: "5b1f3b6effa1b206da200dc4b949375f"
server_key_version: "encryptionkey@0"
I20260927 15:00:56.418012 19764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:56.444437 19764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:56.448606 19764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:56.448865 19764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:56.449498 19764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:00:56.450376 19764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:00:56.450423 19764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:56.450465 19764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:00:56.450486 19764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:56.459551 19764 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33485
I20260927 15:00:56.459586 19892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33485 every 8 connection(s)
I20260927 15:00:56.460493 19764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:00:56.465270 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 19764
I20260927 15:00:56.465380 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:00:56.465613 392 external_mini_cluster.cc:1428] Setting key 59cbdd2101510bf678899d4851b26d53
Sep 27 15:00:56 dist-test-slave-pz0h krb5kdc[19523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 15:00:56.475812 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:56.462453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46045 (local address 127.0.98.62:45941)
0927 15:00:56.462603 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:56.462608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:56.463562 (+ 954us) server_negotiation.cc:407] Connection header received
0927 15:00:56.464848 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:56.464851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:56.464900 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:56.464982 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:56.466396 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:56.467119 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:56.468346 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:56.468647 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:56.468733 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:56.471683 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:56.471687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:56.471689 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:00:56.471718 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:56.473107 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:56.473919 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:56.473923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:56.473924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:56.474011 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:56.474485 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:56.474489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:56.474490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:56.474709 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:56.474856 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:56.475504 (+ 648us) server_negotiation.cc:299] Negotiation successful
0927 15:00:56.475649 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 15:00:56.476931 19895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:56.462889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45941 (local address 127.0.98.2:46045)
0927 15:00:56.463401 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:56.463445 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:56.464638 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:56.465230 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:56.465238 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:56.465672 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:00:56.466271 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:56.466289 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:56.467250 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:56.467253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:56.468206 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:56.468218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:56.468812 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:56.468816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:56.469048 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:56.469735 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:00:56.469752 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:00:56.471443 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:00:56.473266 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:00:56.473274 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:00:56.473334 (+ 60us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:00:56.473782 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:00:56.474103 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:00:56.474107 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:00:56.474109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:00:56.474351 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:00:56.474880 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:00:56.474888 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:00:56.475225 (+ 337us) client_negotiation.cc:769] Sending connection context
0927 15:00:56.475420 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 15:00:56.475773 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":151,"threads_started":1}
I20260927 15:00:56.478279 19893 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45941
I20260927 15:00:56.478523 19893 heartbeater.cc:464] Registering TS with master...
I20260927 15:00:56.479084 19893 heartbeater.cc:511] Master 127.0.98.62:45941 requested a full tablet report, sending...
I20260927 15:00:56.480185 19568 ts_manager.cc:194] Registered new tserver with Master: 44b99506d67d48ae9d54c264a1298bb5 (127.0.98.2:33485)
I20260927 15:00:56.480780 19568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46045
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 15:00:56.491290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45941
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:00:56.606097 19898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:56.606408 19898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:56.606464 19898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:56.610625 19898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:56.610708 19898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:56.610776 19898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:00:56.616304 19898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:56.617462 19898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:56.618487 19898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:00:56.628177 19906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:56.628166 19904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:56.628166 19903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:56.628268 19898 server_base.cc:1034] running on GCE node
I20260927 15:00:56.628831 19898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:56.629511 19898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:56.630684 19898 hybrid_clock.cc:648] HybridClock initialized: now 1790521256630661 us; error 34 us; skew 500 ppm
Sep 27 15:00:56 dist-test-slave-pz0h krb5kdc[19523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:00:56.633625 19898 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 15:00:56.634768 19898 webserver.cc:460] Webserver started at http://127.0.98.3:35427/ using document root <none> and password file <none>
I20260927 15:00:56.635421 19898 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:56.635493 19898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:56.635700 19898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:00:56.637887 19898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-2/data/instance:
uuid: "caca910641fd4b50b1a6bc9db82ab763"
format_stamp: "Formatted at 2026-09-27 15:00:56 on dist-test-slave-pz0h"
server_key: "f43651653558ef061f339bdf36c87521"
server_key_iv: "12c6036114b1bb17d22e932354c66a74"
server_key_version: "encryptionkey@0"
I20260927 15:00:56.638463 19898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "caca910641fd4b50b1a6bc9db82ab763"
format_stamp: "Formatted at 2026-09-27 15:00:56 on dist-test-slave-pz0h"
server_key: "f43651653558ef061f339bdf36c87521"
server_key_iv: "12c6036114b1bb17d22e932354c66a74"
server_key_version: "encryptionkey@0"
I20260927 15:00:56.642365 19898 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260927 15:00:56.645135 19913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:56.646453 19898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 15:00:56.646585 19898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "caca910641fd4b50b1a6bc9db82ab763"
format_stamp: "Formatted at 2026-09-27 15:00:56 on dist-test-slave-pz0h"
server_key: "f43651653558ef061f339bdf36c87521"
server_key_iv: "12c6036114b1bb17d22e932354c66a74"
server_key_version: "encryptionkey@0"
I20260927 15:00:56.646699 19898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:56.670338 19898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:56.673396 19898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:56.673591 19898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:56.674194 19898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:00:56.675087 19898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:00:56.675138 19898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:56.675194 19898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:00:56.675221 19898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:56.683813 19898 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42879
I20260927 15:00:56.683849 20026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42879 every 8 connection(s)
I20260927 15:00:56.684755 19898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:00:56.687781 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 19898
I20260927 15:00:56.687928 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-2/wal/instance
I20260927 15:00:56.688239 392 external_mini_cluster.cc:1428] Setting key de1c7b4f1f72c52c3519b1f51ce25f0b
Sep 27 15:00:56 dist-test-slave-pz0h krb5kdc[19523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:00:56.699199 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:56.686712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49849 (local address 127.0.98.62:45941)
0927 15:00:56.686906 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:56.686911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:56.687902 (+ 991us) server_negotiation.cc:407] Connection header received
0927 15:00:56.689465 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:56.689469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:56.689528 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:56.689613 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:56.691108 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:56.691650 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:56.692844 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:56.693069 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:56.693155 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:56.695695 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:56.695699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:56.695701 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:00:56.695729 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:56.697036 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:56.697574 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:56.697577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:56.697578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:56.697655 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:56.698003 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:56.698005 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:56.698007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:56.698196 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:56.698280 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:56.698871 (+ 591us) server_negotiation.cc:299] Negotiation successful
0927 15:00:56.699022 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 15:00:56.700057 20029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:56.687178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45941 (local address 127.0.98.3:49849)
0927 15:00:56.687699 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:56.687747 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:56.689212 (+ 1465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:56.689870 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:56.689880 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:56.690389 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:00:56.690974 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:56.690993 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:56.691778 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:56.691781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:56.692700 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:56.692711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:56.693225 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:56.693228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:56.693398 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:56.693962 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:00:56.693980 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:00:56.695507 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:00:56.697165 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:00:56.697171 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:00:56.697189 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:00:56.697466 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:00:56.697740 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:00:56.697743 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:00:56.697745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:00:56.697904 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:00:56.698298 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:00:56.698305 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:00:56.698580 (+ 275us) client_negotiation.cc:769] Sending connection context
0927 15:00:56.698755 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 15:00:56.699175 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":141,"threads_started":1}
I20260927 15:00:56.701593 20027 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45941
I20260927 15:00:56.701890 20027 heartbeater.cc:464] Registering TS with master...
I20260927 15:00:56.702517 20027 heartbeater.cc:511] Master 127.0.98.62:45941 requested a full tablet report, sending...
I20260927 15:00:56.703825 19568 ts_manager.cc:194] Registered new tserver with Master: caca910641fd4b50b1a6bc9db82ab763 (127.0.98.3:42879)
I20260927 15:00:56.704430 19568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49849
I20260927 15:00:56.714847 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:00:56.724172 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:56.716926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37300 (local address 127.0.98.62:45941)
0927 15:00:56.717088 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:56.717093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:56.717230 (+ 137us) server_negotiation.cc:407] Connection header received
0927 15:00:56.717361 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:56.717364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:56.717409 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:56.717476 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:56.718200 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:56.718728 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:56.719631 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:56.719847 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:56.719935 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:56.721244 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:56.721248 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:56.721250 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:00:56.721276 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:56.722614 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:56.723092 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:56.723095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:56.723097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:56.723184 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:56.723408 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:56.723410 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:56.723411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:56.723635 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:56.723739 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:56.723902 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 15:00:56.724009 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 15:00:56.727157 19568 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37300:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:00:56.730253 19568 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:00:56.744905 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:56.739097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42879 (local address 127.0.0.1:37060)
0927 15:00:56.739478 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:56.739529 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:56.739657 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:56.740164 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:56.740172 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:56.740199 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:56.740362 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:56.740370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:56.741970 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:56.741974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:56.743257 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:56.743268 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:56.744296 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:56.744300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:56.744444 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:56.744498 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 15:00:56.744591 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 15:00:56.744735 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":50,"thread_start_us":89,"threads_started":1}
I20260927 15:00:56.745075 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:56.739859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33485 (local address 127.0.0.1:42086)
0927 15:00:56.740092 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:56.740104 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:56.740180 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:56.740920 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:56.740924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:56.740942 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:56.741037 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:56.741043 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:56.742368 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:56.742371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:56.743533 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:56.743544 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:56.744645 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:56.744648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:56.744742 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:56.744763 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:00:56.744821 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 15:00:56.744876 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":4,"thread_start_us":113,"threads_started":1}
I20260927 15:00:56.745150 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:56.740041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42086 (local address 127.0.98.2:33485)
0927 15:00:56.740479 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:56.740491 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:56.740512 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:00:56.740584 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:56.740589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:56.740769 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:56.741177 (+ 408us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:56.741338 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:56.742238 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:56.743703 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:56.744542 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:56.744651 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:56.744826 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:56.744884 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 15:00:56.744989 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":132,"threads_started":1}
I20260927 15:00:56.745673 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:56.740186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46733 (local address 127.0.0.1:46310)
0927 15:00:56.740377 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:56.740400 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:56.740477 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:56.741221 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:56.741224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:56.741240 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:56.741326 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:56.741332 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:56.742701 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:56.742704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:56.743835 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:56.743844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:56.744886 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:56.744890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:56.744976 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:56.745467 (+ 491us) client_negotiation.cc:769] Sending connection context
0927 15:00:56.745515 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 15:00:56.745576 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
I20260927 15:00:56.745813 20039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:56.739233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37060 (local address 127.0.98.3:42879)
0927 15:00:56.739677 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:56.739690 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:56.739712 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:00:56.739791 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:56.739796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:56.739973 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:56.740520 (+ 547us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:56.740671 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:56.741823 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:56.743394 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:56.744166 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:56.745369 (+ 1203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:56.745530 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:56.745577 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 15:00:56.745669 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":149,"threads_started":1}
I20260927 15:00:56.746395 20043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:56.740570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46310 (local address 127.0.98.1:46733)
0927 15:00:56.740815 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:56.740822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:56.740843 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:00:56.740913 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:56.740918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:56.741091 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:56.741471 (+ 380us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:56.741598 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:56.742579 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:56.743968 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:56.744755 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:56.745941 (+ 1186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:56.746095 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:56.746145 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 15:00:56.746242 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":60,"threads_started":1}
I20260927 15:00:56.748365 19827 tablet_service.cc:1467] Processing CreateTablet for tablet b4e086fd06344bd29e7b1e8b9c2e1774 (DEFAULT_TABLE table=test-table [id=875ed2c9e5434a05875b6527de1d10ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:00:56.748580 19961 tablet_service.cc:1467] Processing CreateTablet for tablet b4e086fd06344bd29e7b1e8b9c2e1774 (DEFAULT_TABLE table=test-table [id=875ed2c9e5434a05875b6527de1d10ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:00:56.749439 19827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4e086fd06344bd29e7b1e8b9c2e1774. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:00:56.749634 19961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4e086fd06344bd29e7b1e8b9c2e1774. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:00:56.749274 19693 tablet_service.cc:1467] Processing CreateTablet for tablet b4e086fd06344bd29e7b1e8b9c2e1774 (DEFAULT_TABLE table=test-table [id=875ed2c9e5434a05875b6527de1d10ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:00:56.750296 19693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4e086fd06344bd29e7b1e8b9c2e1774. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:00:56.756206 20044 tablet_bootstrap.cc:492] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5: Bootstrap starting.
I20260927 15:00:56.757717 20046 tablet_bootstrap.cc:492] T b4e086fd06344bd29e7b1e8b9c2e1774 P c14239daf5cb4158aba377093d9531f5: Bootstrap starting.
I20260927 15:00:56.758677 20044 tablet_bootstrap.cc:654] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5: Neither blocks nor log segments found. Creating new log.
I20260927 15:00:56.759837 20046 tablet_bootstrap.cc:654] T b4e086fd06344bd29e7b1e8b9c2e1774 P c14239daf5cb4158aba377093d9531f5: Neither blocks nor log segments found. Creating new log.
I20260927 15:00:56.759924 20044 log.cc:824] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5: Log is configured to *not* fsync() on all Append() calls
I20260927 15:00:56.760520 20046 log.cc:824] T b4e086fd06344bd29e7b1e8b9c2e1774 P c14239daf5cb4158aba377093d9531f5: Log is configured to *not* fsync() on all Append() calls
I20260927 15:00:56.761217 20045 tablet_bootstrap.cc:492] T b4e086fd06344bd29e7b1e8b9c2e1774 P caca910641fd4b50b1a6bc9db82ab763: Bootstrap starting.
I20260927 15:00:56.761977 20044 tablet_bootstrap.cc:492] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5: No bootstrap required, opened a new log
I20260927 15:00:56.762197 20044 ts_tablet_manager.cc:1397] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:00:56.762406 20046 tablet_bootstrap.cc:492] T b4e086fd06344bd29e7b1e8b9c2e1774 P c14239daf5cb4158aba377093d9531f5: No bootstrap required, opened a new log
I20260927 15:00:56.762595 20046 ts_tablet_manager.cc:1397] T b4e086fd06344bd29e7b1e8b9c2e1774 P c14239daf5cb4158aba377093d9531f5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 15:00:56.763497 20045 tablet_bootstrap.cc:654] T b4e086fd06344bd29e7b1e8b9c2e1774 P caca910641fd4b50b1a6bc9db82ab763: Neither blocks nor log segments found. Creating new log.
I20260927 15:00:56.764791 20045 log.cc:824] T b4e086fd06344bd29e7b1e8b9c2e1774 P caca910641fd4b50b1a6bc9db82ab763: Log is configured to *not* fsync() on all Append() calls
I20260927 15:00:56.766384 20044 raft_consensus.cc:348] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caca910641fd4b50b1a6bc9db82ab763" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42879 } } peers { permanent_uuid: "44b99506d67d48ae9d54c264a1298bb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33485 } } peers { permanent_uuid: "c14239daf5cb4158aba377093d9531f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46733 } }
I20260927 15:00:56.766721 20044 raft_consensus.cc:374] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:00:56.766785 20044 raft_consensus.cc:729] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44b99506d67d48ae9d54c264a1298bb5, State: Initialized, Role: FOLLOWER
I20260927 15:00:56.766863 20045 tablet_bootstrap.cc:492] T b4e086fd06344bd29e7b1e8b9c2e1774 P caca910641fd4b50b1a6bc9db82ab763: No bootstrap required, opened a new log
I20260927 15:00:56.767089 20045 ts_tablet_manager.cc:1397] T b4e086fd06344bd29e7b1e8b9c2e1774 P caca910641fd4b50b1a6bc9db82ab763: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:00:56.767087 20046 raft_consensus.cc:348] T b4e086fd06344bd29e7b1e8b9c2e1774 P c14239daf5cb4158aba377093d9531f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caca910641fd4b50b1a6bc9db82ab763" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42879 } } peers { permanent_uuid: "44b99506d67d48ae9d54c264a1298bb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33485 } } peers { permanent_uuid: "c14239daf5cb4158aba377093d9531f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46733 } }
I20260927 15:00:56.767356 20046 raft_consensus.cc:374] T b4e086fd06344bd29e7b1e8b9c2e1774 P c14239daf5cb4158aba377093d9531f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:00:56.767318 20044 consensus_queue.cc:260] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caca910641fd4b50b1a6bc9db82ab763" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42879 } } peers { permanent_uuid: "44b99506d67d48ae9d54c264a1298bb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33485 } } peers { permanent_uuid: "c14239daf5cb4158aba377093d9531f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46733 } }
I20260927 15:00:56.767421 20046 raft_consensus.cc:729] T b4e086fd06344bd29e7b1e8b9c2e1774 P c14239daf5cb4158aba377093d9531f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c14239daf5cb4158aba377093d9531f5, State: Initialized, Role: FOLLOWER
I20260927 15:00:56.767849 20046 consensus_queue.cc:260] T b4e086fd06344bd29e7b1e8b9c2e1774 P c14239daf5cb4158aba377093d9531f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caca910641fd4b50b1a6bc9db82ab763" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42879 } } peers { permanent_uuid: "44b99506d67d48ae9d54c264a1298bb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33485 } } peers { permanent_uuid: "c14239daf5cb4158aba377093d9531f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46733 } }
I20260927 15:00:56.768448 20044 ts_tablet_manager.cc:1428] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260927 15:00:56.768584 19759 heartbeater.cc:503] Master 127.0.98.62:45941 was elected leader, sending a full tablet report...
I20260927 15:00:56.768800 19893 heartbeater.cc:503] Master 127.0.98.62:45941 was elected leader, sending a full tablet report...
I20260927 15:00:56.768807 20046 ts_tablet_manager.cc:1428] T b4e086fd06344bd29e7b1e8b9c2e1774 P c14239daf5cb4158aba377093d9531f5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 15:00:56.771631 20045 raft_consensus.cc:348] T b4e086fd06344bd29e7b1e8b9c2e1774 P caca910641fd4b50b1a6bc9db82ab763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caca910641fd4b50b1a6bc9db82ab763" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42879 } } peers { permanent_uuid: "44b99506d67d48ae9d54c264a1298bb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33485 } } peers { permanent_uuid: "c14239daf5cb4158aba377093d9531f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46733 } }
I20260927 15:00:56.771929 20045 raft_consensus.cc:374] T b4e086fd06344bd29e7b1e8b9c2e1774 P caca910641fd4b50b1a6bc9db82ab763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:00:56.772001 20045 raft_consensus.cc:729] T b4e086fd06344bd29e7b1e8b9c2e1774 P caca910641fd4b50b1a6bc9db82ab763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caca910641fd4b50b1a6bc9db82ab763, State: Initialized, Role: FOLLOWER
I20260927 15:00:56.772914 20045 consensus_queue.cc:260] T b4e086fd06344bd29e7b1e8b9c2e1774 P caca910641fd4b50b1a6bc9db82ab763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caca910641fd4b50b1a6bc9db82ab763" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42879 } } peers { permanent_uuid: "44b99506d67d48ae9d54c264a1298bb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33485 } } peers { permanent_uuid: "c14239daf5cb4158aba377093d9531f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46733 } }
I20260927 15:00:56.773613 20027 heartbeater.cc:503] Master 127.0.98.62:45941 was elected leader, sending a full tablet report...
I20260927 15:00:56.774029 20045 ts_tablet_manager.cc:1428] T b4e086fd06344bd29e7b1e8b9c2e1774 P caca910641fd4b50b1a6bc9db82ab763: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20260927 15:00:56.935806 20028 tablet.cc:2369] T b4e086fd06344bd29e7b1e8b9c2e1774 P caca910641fd4b50b1a6bc9db82ab763: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:00:56.961719 19894 tablet.cc:2369] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:00:56.972005 19760 tablet.cc:2369] T b4e086fd06344bd29e7b1e8b9c2e1774 P c14239daf5cb4158aba377093d9531f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:00:57.098402 20050 raft_consensus.cc:482] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:00:57.098644 20050 raft_consensus.cc:504] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caca910641fd4b50b1a6bc9db82ab763" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42879 } } peers { permanent_uuid: "44b99506d67d48ae9d54c264a1298bb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33485 } } peers { permanent_uuid: "c14239daf5cb4158aba377093d9531f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46733 } }
I20260927 15:00:57.099812 20050 leader_election.cc:288] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers caca910641fd4b50b1a6bc9db82ab763 (127.0.98.3:42879), c14239daf5cb4158aba377093d9531f5 (127.0.98.1:46733)
I20260927 15:00:57.104086 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:57.100499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46733 (local address 127.0.98.2:44405)
0927 15:00:57.100695 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:57.100709 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:57.100824 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:57.101039 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:57.101042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:57.101057 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:57.101166 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:57.101171 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:57.101853 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:57.101856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:57.102762 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:57.102773 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:57.103602 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:57.103606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:57.103744 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:57.103762 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 15:00:57.103833 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 15:00:57.103919 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":38,"thread_start_us":66,"threads_started":1}
I20260927 15:00:57.104074 20043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:57.100671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44405 (local address 127.0.98.1:46733)
0927 15:00:57.100819 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:57.100823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:57.100838 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:00:57.100917 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:57.100920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:57.100960 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:57.101019 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:57.101278 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:57.101754 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:57.102901 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:57.103468 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:57.103585 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:57.103647 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:57.103858 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 15:00:57.103951 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 15:00:57.104357 20039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:57.100232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37777 (local address 127.0.98.3:42879)
0927 15:00:57.100419 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:57.100424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:57.100524 (+ 100us) server_negotiation.cc:407] Connection header received
0927 15:00:57.100707 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:57.100711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:57.100765 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:57.100811 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:57.101166 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:57.101672 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:57.102915 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:57.103385 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:57.103476 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:57.103542 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:57.104105 (+ 563us) server_negotiation.cc:299] Negotiation successful
0927 15:00:57.104213 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 15:00:57.104691 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:57.100103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42879 (local address 127.0.98.2:37777)
0927 15:00:57.100420 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:57.100437 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:57.100571 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:57.100935 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:57.100938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:57.100953 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:57.101054 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:57.101058 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:57.101786 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:57.101789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:57.102754 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:57.102766 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:57.103511 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:57.103514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:57.103660 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:57.103681 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:00:57.103944 (+ 263us) client_negotiation.cc:240] Negotiation successful
0927 15:00:57.104560 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":85,"thread_start_us":123,"threads_started":1}
I20260927 15:00:57.105070 19713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4e086fd06344bd29e7b1e8b9c2e1774" candidate_uuid: "44b99506d67d48ae9d54c264a1298bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c14239daf5cb4158aba377093d9531f5" is_pre_election: true
I20260927 15:00:57.105332 19981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4e086fd06344bd29e7b1e8b9c2e1774" candidate_uuid: "44b99506d67d48ae9d54c264a1298bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caca910641fd4b50b1a6bc9db82ab763" is_pre_election: true
I20260927 15:00:57.105407 19713 raft_consensus.cc:2454] T b4e086fd06344bd29e7b1e8b9c2e1774 P c14239daf5cb4158aba377093d9531f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44b99506d67d48ae9d54c264a1298bb5 in term 0.
I20260927 15:00:57.105590 19981 raft_consensus.cc:2454] T b4e086fd06344bd29e7b1e8b9c2e1774 P caca910641fd4b50b1a6bc9db82ab763 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44b99506d67d48ae9d54c264a1298bb5 in term 0.
I20260927 15:00:57.105932 19781 leader_election.cc:302] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44b99506d67d48ae9d54c264a1298bb5, c14239daf5cb4158aba377093d9531f5; no voters:
I20260927 15:00:57.106215 20050 raft_consensus.cc:2781] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:00:57.106308 20050 raft_consensus.cc:482] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:00:57.106364 20050 raft_consensus.cc:3037] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:00:57.107514 20050 raft_consensus.cc:504] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caca910641fd4b50b1a6bc9db82ab763" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42879 } } peers { permanent_uuid: "44b99506d67d48ae9d54c264a1298bb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33485 } } peers { permanent_uuid: "c14239daf5cb4158aba377093d9531f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46733 } }
I20260927 15:00:57.107904 20050 leader_election.cc:288] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5 [CANDIDATE]: Term 1 election: Requested vote from peers caca910641fd4b50b1a6bc9db82ab763 (127.0.98.3:42879), c14239daf5cb4158aba377093d9531f5 (127.0.98.1:46733)
I20260927 15:00:57.108410 19981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4e086fd06344bd29e7b1e8b9c2e1774" candidate_uuid: "44b99506d67d48ae9d54c264a1298bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caca910641fd4b50b1a6bc9db82ab763"
I20260927 15:00:57.108459 19713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4e086fd06344bd29e7b1e8b9c2e1774" candidate_uuid: "44b99506d67d48ae9d54c264a1298bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c14239daf5cb4158aba377093d9531f5"
I20260927 15:00:57.108556 19981 raft_consensus.cc:3037] T b4e086fd06344bd29e7b1e8b9c2e1774 P caca910641fd4b50b1a6bc9db82ab763 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:00:57.108582 19713 raft_consensus.cc:3037] T b4e086fd06344bd29e7b1e8b9c2e1774 P c14239daf5cb4158aba377093d9531f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:00:57.109773 19981 raft_consensus.cc:2454] T b4e086fd06344bd29e7b1e8b9c2e1774 P caca910641fd4b50b1a6bc9db82ab763 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44b99506d67d48ae9d54c264a1298bb5 in term 1.
I20260927 15:00:57.109810 19713 raft_consensus.cc:2454] T b4e086fd06344bd29e7b1e8b9c2e1774 P c14239daf5cb4158aba377093d9531f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44b99506d67d48ae9d54c264a1298bb5 in term 1.
I20260927 15:00:57.110173 19781 leader_election.cc:302] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44b99506d67d48ae9d54c264a1298bb5, caca910641fd4b50b1a6bc9db82ab763; no voters:
I20260927 15:00:57.110495 20050 raft_consensus.cc:2781] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:00:57.110778 20050 raft_consensus.cc:686] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5 [term 1 LEADER]: Becoming Leader. State: Replica: 44b99506d67d48ae9d54c264a1298bb5, State: Running, Role: LEADER
I20260927 15:00:57.111200 20050 consensus_queue.cc:237] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caca910641fd4b50b1a6bc9db82ab763" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42879 } } peers { permanent_uuid: "44b99506d67d48ae9d54c264a1298bb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33485 } } peers { permanent_uuid: "c14239daf5cb4158aba377093d9531f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46733 } }
I20260927 15:00:57.114679 19567 catalog_manager.cc:5495] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44b99506d67d48ae9d54c264a1298bb5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "44b99506d67d48ae9d54c264a1298bb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caca910641fd4b50b1a6bc9db82ab763" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44b99506d67d48ae9d54c264a1298bb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c14239daf5cb4158aba377093d9531f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46733 } health_report { overall_health: UNKNOWN } } }
I20260927 15:00:57.208413 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:57.204777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42090 (local address 127.0.98.2:33485)
0927 15:00:57.204896 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:57.204900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:57.204914 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:00:57.205018 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:57.205021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:57.205077 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:57.205175 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:00:57.205376 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:57.205886 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:57.206830 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:57.207076 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:57.207168 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:57.207567 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:00:57.208033 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:00:57.208130 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:57.208227 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 15:00:57.208286 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 15:00:57.215107 19713 raft_consensus.cc:1261] T b4e086fd06344bd29e7b1e8b9c2e1774 P c14239daf5cb4158aba377093d9531f5 [term 1 FOLLOWER]: Refusing update from remote peer 44b99506d67d48ae9d54c264a1298bb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:00:57.215139 19981 raft_consensus.cc:1261] T b4e086fd06344bd29e7b1e8b9c2e1774 P caca910641fd4b50b1a6bc9db82ab763 [term 1 FOLLOWER]: Refusing update from remote peer 44b99506d67d48ae9d54c264a1298bb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:00:57.216373 20055 consensus_queue.cc:1035] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c14239daf5cb4158aba377093d9531f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:00:57.216740 20050 consensus_queue.cc:1035] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "caca910641fd4b50b1a6bc9db82ab763" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:00:57.226626 20058 mvcc.cc:205] Tried to move back new op lower bound from 7333975069544939520 to 7333975069130817536. Current Snapshot: MvccSnapshot[applied={T|T < 7333975069544939520}]
I20260927 15:00:57.228560 20060 mvcc.cc:205] Tried to move back new op lower bound from 7333975069544939520 to 7333975069130817536. Current Snapshot: MvccSnapshot[applied={T|T < 7333975069544939520}]
I20260927 15:00:57.230379 20059 mvcc.cc:205] Tried to move back new op lower bound from 7333975069544939520 to 7333975069130817536. Current Snapshot: MvccSnapshot[applied={T|T < 7333975069544939520 or (T in {7333975069544939520})}]
I20260927 15:00:57.240667 19566 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:00:57.240903 19566 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:00:57.244381 19566 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f601f7a0557943638ca67afbd2fa6490: Sending DeleteTablet for 3 replicas of tablet b4e086fd06344bd29e7b1e8b9c2e1774
I20260927 15:00:57.245301 19961 tablet_service.cc:1514] Processing DeleteTablet for tablet b4e086fd06344bd29e7b1e8b9c2e1774 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:00:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37060
I20260927 15:00:57.245397 19827 tablet_service.cc:1514] Processing DeleteTablet for tablet b4e086fd06344bd29e7b1e8b9c2e1774 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:00:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42086
I20260927 15:00:57.245853 19693 tablet_service.cc:1514] Processing DeleteTablet for tablet b4e086fd06344bd29e7b1e8b9c2e1774 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:00:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46310
I20260927 15:00:57.246186 20068 tablet_replica.cc:330] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5: stopping tablet replica
I20260927 15:00:57.246389 20069 tablet_replica.cc:330] T b4e086fd06344bd29e7b1e8b9c2e1774 P c14239daf5cb4158aba377093d9531f5: stopping tablet replica
I20260927 15:00:57.246660 20068 raft_consensus.cc:2229] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:00:57.246897 20067 tablet_replica.cc:330] T b4e086fd06344bd29e7b1e8b9c2e1774 P caca910641fd4b50b1a6bc9db82ab763: stopping tablet replica
I20260927 15:00:57.247203 20067 raft_consensus.cc:2229] T b4e086fd06344bd29e7b1e8b9c2e1774 P caca910641fd4b50b1a6bc9db82ab763 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:00:57.247326 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 19630
I20260927 15:00:57.247431 20068 raft_consensus.cc:2258] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:00:57.249728 20067 raft_consensus.cc:2258] T b4e086fd06344bd29e7b1e8b9c2e1774 P caca910641fd4b50b1a6bc9db82ab763 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:00:57.250680 20068 ts_tablet_manager.cc:1905] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:00:57.251156 20067 ts_tablet_manager.cc:1905] T b4e086fd06344bd29e7b1e8b9c2e1774 P caca910641fd4b50b1a6bc9db82ab763: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:00:57.253706 20068 ts_tablet_manager.cc:1918] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:00:57.253805 20067 ts_tablet_manager.cc:1918] T b4e086fd06344bd29e7b1e8b9c2e1774 P caca910641fd4b50b1a6bc9db82ab763: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:00:57.253832 20068 log.cc:1196] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-1/wal/wals/b4e086fd06344bd29e7b1e8b9c2e1774
I20260927 15:00:57.253901 20067 log.cc:1196] T b4e086fd06344bd29e7b1e8b9c2e1774 P caca910641fd4b50b1a6bc9db82ab763: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790521004237705-392-0/minicluster-data/ts-2/wal/wals/b4e086fd06344bd29e7b1e8b9c2e1774
I20260927 15:00:57.254132 20068 ts_tablet_manager.cc:1939] T b4e086fd06344bd29e7b1e8b9c2e1774 P 44b99506d67d48ae9d54c264a1298bb5: Deleting consensus metadata
I20260927 15:00:57.254184 20067 ts_tablet_manager.cc:1939] T b4e086fd06344bd29e7b1e8b9c2e1774 P caca910641fd4b50b1a6bc9db82ab763: Deleting consensus metadata
I20260927 15:00:57.255573 19553 catalog_manager.cc:4841] TS caca910641fd4b50b1a6bc9db82ab763 (127.0.98.3:42879): tablet b4e086fd06344bd29e7b1e8b9c2e1774 (table test-table [id=875ed2c9e5434a05875b6527de1d10ce]) successfully deleted
I20260927 15:00:57.255823 19553 catalog_manager.cc:4841] TS 44b99506d67d48ae9d54c264a1298bb5 (127.0.98.2:33485): tablet b4e086fd06344bd29e7b1e8b9c2e1774 (table test-table [id=875ed2c9e5434a05875b6527de1d10ce]) successfully deleted
W20260927 15:00:57.256280 19553 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46733 (error 108)
I20260927 15:00:57.256518 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 19764
I20260927 15:00:57.258415 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:57.257657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46733 (local address 127.0.0.1:46316)
0927 15:00:57.258248 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:57.258325 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":69,"threads_started":1}
W20260927 15:00:57.258574 19553 catalog_manager.cc:4584] TS c14239daf5cb4158aba377093d9531f5 (127.0.98.1:46733): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b4e086fd06344bd29e7b1e8b9c2e1774: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46733: connect: Connection refused (error 111)
I20260927 15:00:57.262040 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 19898
I20260927 15:00:57.267630 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 19536
2026-09-27T15:00:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2602 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T15:00:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:00:57Z Disabled control of system clock
I20260927 15:00:57.297227 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41471
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:00:57.414889 20075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:57.415201 20075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:57.415238 20075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:57.419582 20075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:00:57.419665 20075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:57.419679 20075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:57.419695 20075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:00:57.419710 20075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:00:57.424922 20075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41471
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:57.426108 20075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:57.427569 20075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:00:57.436070 20081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:57.436033 20080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:57.436365 20075 server_base.cc:1034] running on GCE node
W20260927 15:00:57.436044 20083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:57.437207 20075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:57.438361 20075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:57.439535 20075 hybrid_clock.cc:648] HybridClock initialized: now 1790521257439507 us; error 47 us; skew 500 ppm
I20260927 15:00:57.441627 20075 webserver.cc:460] Webserver started at http://127.0.98.62:38723/ using document root <none> and password file <none>
I20260927 15:00:57.442243 20075 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:57.442315 20075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:57.442519 20075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:00:57.444784 20075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "b128e5107ec64d808f896b1a16fc5933"
format_stamp: "Formatted at 2026-09-27 15:00:57 on dist-test-slave-pz0h"
server_key: "72d127503b7834ad8863dd6a0a9421fc"
server_key_iv: "a1c82e594c5200d8ef437f1f3a2d9b9a"
server_key_version: "encryptionkey@0"
I20260927 15:00:57.445371 20075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "b128e5107ec64d808f896b1a16fc5933"
format_stamp: "Formatted at 2026-09-27 15:00:57 on dist-test-slave-pz0h"
server_key: "72d127503b7834ad8863dd6a0a9421fc"
server_key_iv: "a1c82e594c5200d8ef437f1f3a2d9b9a"
server_key_version: "encryptionkey@0"
I20260927 15:00:57.449302 20075 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:00:57.451946 20089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:57.453225 20075 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:00:57.453361 20075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "b128e5107ec64d808f896b1a16fc5933"
format_stamp: "Formatted at 2026-09-27 15:00:57 on dist-test-slave-pz0h"
server_key: "72d127503b7834ad8863dd6a0a9421fc"
server_key_iv: "a1c82e594c5200d8ef437f1f3a2d9b9a"
server_key_version: "encryptionkey@0"
I20260927 15:00:57.453474 20075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:57.483093 20075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:57.483857 20075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:57.484037 20075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:57.491132 20075 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41471
I20260927 15:00:57.491158 20141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41471 every 8 connection(s)
I20260927 15:00:57.492295 20075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:00:57.494324 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 20075
I20260927 15:00:57.494467 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 15:00:57.494760 392 external_mini_cluster.cc:1428] Setting key 58fb0d7a11521e87a249f74020be0bd6
I20260927 15:00:57.496536 20142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:00:57.502815 20142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b128e5107ec64d808f896b1a16fc5933: Bootstrap starting.
I20260927 15:00:57.505190 20142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b128e5107ec64d808f896b1a16fc5933: Neither blocks nor log segments found. Creating new log.
I20260927 15:00:57.505924 20142 log.cc:824] T 00000000000000000000000000000000 P b128e5107ec64d808f896b1a16fc5933: Log is configured to *not* fsync() on all Append() calls
I20260927 15:00:57.508131 20142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b128e5107ec64d808f896b1a16fc5933: No bootstrap required, opened a new log
I20260927 15:00:57.510543 20145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:57.496389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46046 (local address 127.0.98.62:41471)
0927 15:00:57.497184 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:57.497200 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:57.497240 (+ 40us) server_negotiation.cc:407] Connection header received
0927 15:00:57.497932 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:57.497950 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:57.498381 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:57.498728 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:57.499224 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:57.500221 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:57.501118 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:57.501608 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:57.501759 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:57.508692 (+ 6933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:57.508710 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:57.508727 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:00:57.509140 (+ 413us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:57.509209 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:00:57.509233 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:57.509328 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:57.509486 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 15:00:57.509789 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":148,"threads_started":1}
I20260927 15:00:57.512167 20142 raft_consensus.cc:348] T 00000000000000000000000000000000 P b128e5107ec64d808f896b1a16fc5933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b128e5107ec64d808f896b1a16fc5933" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41471 } }
I20260927 15:00:57.512428 20142 raft_consensus.cc:374] T 00000000000000000000000000000000 P b128e5107ec64d808f896b1a16fc5933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:00:57.512465 20142 raft_consensus.cc:729] T 00000000000000000000000000000000 P b128e5107ec64d808f896b1a16fc5933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b128e5107ec64d808f896b1a16fc5933, State: Initialized, Role: FOLLOWER
I20260927 15:00:57.512954 20142 consensus_queue.cc:260] T 00000000000000000000000000000000 P b128e5107ec64d808f896b1a16fc5933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b128e5107ec64d808f896b1a16fc5933" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41471 } }
I20260927 15:00:57.513108 20142 raft_consensus.cc:388] T 00000000000000000000000000000000 P b128e5107ec64d808f896b1a16fc5933 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:00:57.513291 20142 raft_consensus.cc:482] T 00000000000000000000000000000000 P b128e5107ec64d808f896b1a16fc5933 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:00:57.513603 20142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b128e5107ec64d808f896b1a16fc5933 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:00:57.514861 20142 raft_consensus.cc:504] T 00000000000000000000000000000000 P b128e5107ec64d808f896b1a16fc5933 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b128e5107ec64d808f896b1a16fc5933" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41471 } }
I20260927 15:00:57.515564 20142 leader_election.cc:302] T 00000000000000000000000000000000 P b128e5107ec64d808f896b1a16fc5933 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b128e5107ec64d808f896b1a16fc5933; no voters:
I20260927 15:00:57.515964 20142 leader_election.cc:288] T 00000000000000000000000000000000 P b128e5107ec64d808f896b1a16fc5933 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:00:57.516379 20147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b128e5107ec64d808f896b1a16fc5933 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:00:57.516592 20147 raft_consensus.cc:686] T 00000000000000000000000000000000 P b128e5107ec64d808f896b1a16fc5933 [term 1 LEADER]: Becoming Leader. State: Replica: b128e5107ec64d808f896b1a16fc5933, State: Running, Role: LEADER
I20260927 15:00:57.517004 20147 consensus_queue.cc:237] T 00000000000000000000000000000000 P b128e5107ec64d808f896b1a16fc5933 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b128e5107ec64d808f896b1a16fc5933" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41471 } }
I20260927 15:00:57.517606 20142 sys_catalog.cc:564] T 00000000000000000000000000000000 P b128e5107ec64d808f896b1a16fc5933 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:00:57.518843 20149 sys_catalog.cc:455] T 00000000000000000000000000000000 P b128e5107ec64d808f896b1a16fc5933 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b128e5107ec64d808f896b1a16fc5933. Latest consensus state: current_term: 1 leader_uuid: "b128e5107ec64d808f896b1a16fc5933" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b128e5107ec64d808f896b1a16fc5933" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41471 } } }
I20260927 15:00:57.518863 20148 sys_catalog.cc:455] T 00000000000000000000000000000000 P b128e5107ec64d808f896b1a16fc5933 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b128e5107ec64d808f896b1a16fc5933" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b128e5107ec64d808f896b1a16fc5933" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41471 } } }
I20260927 15:00:57.519014 20149 sys_catalog.cc:458] T 00000000000000000000000000000000 P b128e5107ec64d808f896b1a16fc5933 [sys.catalog]: This master's current role is: LEADER
I20260927 15:00:57.519017 20148 sys_catalog.cc:458] T 00000000000000000000000000000000 P b128e5107ec64d808f896b1a16fc5933 [sys.catalog]: This master's current role is: LEADER
I20260927 15:00:57.519435 20156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:00:57.522318 20156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:00:57.528496 20156 catalog_manager.cc:1347] Generated new cluster ID: 372631b8f0474e5a8212d3950b1b26ad
I20260927 15:00:57.528584 20156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:00:57.535243 20156 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:00:57.536558 20156 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:00:57.550210 20156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b128e5107ec64d808f896b1a16fc5933: Generated new TSK 0
I20260927 15:00:57.551211 20156 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:00:57.554703 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41471
--builtin_ntp_servers=127.0.98.20:46499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:00:57.668345 20166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:57.668658 20166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:57.668707 20166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:57.675771 20166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:57.675879 20166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:57.675948 20166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 15:00:57.682019 20166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:57.683168 20166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:57.684508 20166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:00:57.694661 20172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:57.694682 20174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:57.694660 20171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:57.695097 20166 server_base.cc:1034] running on GCE node
I20260927 15:00:57.695523 20166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:57.696187 20166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:57.697366 20166 hybrid_clock.cc:648] HybridClock initialized: now 1790521257697347 us; error 33 us; skew 500 ppm
I20260927 15:00:57.699433 20166 webserver.cc:460] Webserver started at http://127.0.98.1:37735/ using document root <none> and password file <none>
I20260927 15:00:57.700037 20166 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:57.700102 20166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:57.700296 20166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:00:57.702509 20166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-0/data/instance:
uuid: "94cc1cc8d75343eca1f4fcd9468537e5"
format_stamp: "Formatted at 2026-09-27 15:00:57 on dist-test-slave-pz0h"
server_key: "7e18346edb579f6a2b95b24558a5c55a"
server_key_iv: "f7dd9b727166a75daf7c5c9840ab8b4b"
server_key_version: "encryptionkey@0"
I20260927 15:00:57.703119 20166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-0/wal/instance:
uuid: "94cc1cc8d75343eca1f4fcd9468537e5"
format_stamp: "Formatted at 2026-09-27 15:00:57 on dist-test-slave-pz0h"
server_key: "7e18346edb579f6a2b95b24558a5c55a"
server_key_iv: "f7dd9b727166a75daf7c5c9840ab8b4b"
server_key_version: "encryptionkey@0"
I20260927 15:00:57.707118 20166 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:00:57.710119 20180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:57.711896 20166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:00:57.712064 20166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-0/wal
uuid: "94cc1cc8d75343eca1f4fcd9468537e5"
format_stamp: "Formatted at 2026-09-27 15:00:57 on dist-test-slave-pz0h"
server_key: "7e18346edb579f6a2b95b24558a5c55a"
server_key_iv: "f7dd9b727166a75daf7c5c9840ab8b4b"
server_key_version: "encryptionkey@0"
I20260927 15:00:57.712270 20166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:57.739142 20166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:57.739926 20166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:57.740106 20166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:57.740720 20166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:00:57.741930 20166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:00:57.741992 20166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:57.742059 20166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:00:57.742087 20166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:57.752359 20166 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41599
I20260927 15:00:57.752380 20293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41599 every 8 connection(s)
I20260927 15:00:57.753386 20166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-0/data/info.pb
I20260927 15:00:57.761801 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 20166
I20260927 15:00:57.761933 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-0/wal/instance
I20260927 15:00:57.762269 392 external_mini_cluster.cc:1428] Setting key 54321e44f17db54001bf986f728fef70
I20260927 15:00:57.764490 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41471
--builtin_ntp_servers=127.0.98.20:46499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:00:57.765028 20145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:57.755944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50419 (local address 127.0.98.62:41471)
0927 15:00:57.756124 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:57.756129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:57.757380 (+ 1251us) server_negotiation.cc:407] Connection header received
0927 15:00:57.758951 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:57.758955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:57.759027 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:57.759122 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:57.760301 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:57.760961 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:57.762495 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:57.762922 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:57.763044 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:57.764418 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:57.764422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:57.764426 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:00:57.764442 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:57.764468 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:00:57.764479 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:57.764565 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:57.764783 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 15:00:57.764873 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 15:00:57.765947 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:57.756374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41471 (local address 127.0.98.1:50419)
0927 15:00:57.757232 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:57.757279 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:57.758658 (+ 1379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:57.759328 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:57.759337 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:57.759442 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:00:57.760014 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:57.760030 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:57.761197 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:57.761201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:57.762316 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:57.762331 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:57.763142 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:57.763147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:57.763377 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:57.764108 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:00:57.764110 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:00:57.764159 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:00:57.764243 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:00:57.764597 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:00:57.764607 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:00:57.764614 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 15:00:57.764768 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 15:00:57.765049 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":131,"threads_started":1}
I20260927 15:00:57.767315 20294 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41471
I20260927 15:00:57.767601 20294 heartbeater.cc:464] Registering TS with master...
I20260927 15:00:57.768289 20294 heartbeater.cc:511] Master 127.0.98.62:41471 requested a full tablet report, sending...
I20260927 15:00:57.770053 20106 ts_manager.cc:194] Registered new tserver with Master: 94cc1cc8d75343eca1f4fcd9468537e5 (127.0.98.1:41599)
I20260927 15:00:57.771299 20106 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:50419
W20260927 15:00:57.883889 20297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:57.884222 20297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:57.884261 20297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:57.889376 20297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:57.889459 20297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:57.889533 20297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 15:00:57.895090 20297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:57.896265 20297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:57.897677 20297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:00:57.907759 20305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:57.908023 20297 server_base.cc:1034] running on GCE node
W20260927 15:00:57.907855 20303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:57.907856 20302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:57.908732 20297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:57.909399 20297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:57.910761 20297 hybrid_clock.cc:648] HybridClock initialized: now 1790521257910676 us; error 70 us; skew 500 ppm
I20260927 15:00:57.913239 20297 webserver.cc:460] Webserver started at http://127.0.98.2:43875/ using document root <none> and password file <none>
I20260927 15:00:57.913904 20297 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:57.913981 20297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:57.914187 20297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:00:57.916531 20297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-1/data/instance:
uuid: "cb17e5fc3480458b824f71c6a47aeaca"
format_stamp: "Formatted at 2026-09-27 15:00:57 on dist-test-slave-pz0h"
server_key: "e5751dda8d1feca2c78e119e636ac535"
server_key_iv: "2df655d31df6041013632e00642e7ed3"
server_key_version: "encryptionkey@0"
I20260927 15:00:57.917157 20297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cb17e5fc3480458b824f71c6a47aeaca"
format_stamp: "Formatted at 2026-09-27 15:00:57 on dist-test-slave-pz0h"
server_key: "e5751dda8d1feca2c78e119e636ac535"
server_key_iv: "2df655d31df6041013632e00642e7ed3"
server_key_version: "encryptionkey@0"
I20260927 15:00:57.921396 20297 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:00:57.924203 20311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:57.925503 20297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:00:57.925617 20297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-1/wal
uuid: "cb17e5fc3480458b824f71c6a47aeaca"
format_stamp: "Formatted at 2026-09-27 15:00:57 on dist-test-slave-pz0h"
server_key: "e5751dda8d1feca2c78e119e636ac535"
server_key_iv: "2df655d31df6041013632e00642e7ed3"
server_key_version: "encryptionkey@0"
I20260927 15:00:57.925745 20297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:57.944777 20297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:57.945515 20297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:57.945677 20297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:57.946264 20297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:00:57.947365 20297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:00:57.947417 20297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:57.947460 20297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:00:57.947487 20297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:57.958123 20297 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42643
I20260927 15:00:57.958204 20424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42643 every 8 connection(s)
I20260927 15:00:57.959539 20297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-1/data/info.pb
I20260927 15:00:57.962669 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 20297
I20260927 15:00:57.962819 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-1/wal/instance
I20260927 15:00:57.963085 392 external_mini_cluster.cc:1428] Setting key cf5f37f0a735c688eda43bb44940ef1f
I20260927 15:00:57.965078 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41471
--builtin_ntp_servers=127.0.98.20:46499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:00:57.970772 20145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:57.961832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52493 (local address 127.0.98.62:41471)
0927 15:00:57.962014 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:57.962018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:57.963275 (+ 1257us) server_negotiation.cc:407] Connection header received
0927 15:00:57.964697 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:57.964700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:57.964758 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:57.964847 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:57.965842 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:57.966534 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:57.968293 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:57.968519 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:57.968612 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:57.970088 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:57.970093 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:57.970095 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:00:57.970119 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:57.970163 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:00:57.970176 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:57.970293 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:57.970553 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 15:00:57.970639 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 15:00:57.972044 20427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:57.962247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41471 (local address 127.0.98.2:52493)
0927 15:00:57.963095 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:57.963152 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:57.964470 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:57.965007 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:57.965015 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:57.965112 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:00:57.965696 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:57.965712 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:57.966697 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:57.966702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:57.968126 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:57.968139 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:57.968686 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:57.968690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:57.968912 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:57.969618 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:00:57.969620 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:00:57.969683 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:00:57.969787 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:00:57.970318 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:00:57.970327 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:00:57.970334 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 15:00:57.970494 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 15:00:57.970887 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":153,"threads_started":1}
I20260927 15:00:57.973649 20425 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41471
I20260927 15:00:57.973917 20425 heartbeater.cc:464] Registering TS with master...
I20260927 15:00:57.974503 20425 heartbeater.cc:511] Master 127.0.98.62:41471 requested a full tablet report, sending...
I20260927 15:00:57.975754 20106 ts_manager.cc:194] Registered new tserver with Master: cb17e5fc3480458b824f71c6a47aeaca (127.0.98.2:42643)
I20260927 15:00:57.976585 20106 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:52493
W20260927 15:00:58.082392 20428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:58.082695 20428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:58.082736 20428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:58.087041 20428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:58.087119 20428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:58.087181 20428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 15:00:58.092298 20428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:58.093487 20428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:58.094952 20428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:00:58.104928 20436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:58.104916 20433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:58.104899 20434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:58.105252 20428 server_base.cc:1034] running on GCE node
I20260927 15:00:58.105700 20428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:58.106376 20428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:58.107576 20428 hybrid_clock.cc:648] HybridClock initialized: now 1790521258107555 us; error 40 us; skew 500 ppm
I20260927 15:00:58.110174 20428 webserver.cc:460] Webserver started at http://127.0.98.3:43709/ using document root <none> and password file <none>
I20260927 15:00:58.110844 20428 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:58.110929 20428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:58.111155 20428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:00:58.113479 20428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-2/data/instance:
uuid: "a9c49b0da76d4d21a5baa34878ca2a6a"
format_stamp: "Formatted at 2026-09-27 15:00:58 on dist-test-slave-pz0h"
server_key: "89a123d1b669ed1e964b55d677c7b3f4"
server_key_iv: "7bd114fe0d00ce6d162568acda0356bb"
server_key_version: "encryptionkey@0"
I20260927 15:00:58.114094 20428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a9c49b0da76d4d21a5baa34878ca2a6a"
format_stamp: "Formatted at 2026-09-27 15:00:58 on dist-test-slave-pz0h"
server_key: "89a123d1b669ed1e964b55d677c7b3f4"
server_key_iv: "7bd114fe0d00ce6d162568acda0356bb"
server_key_version: "encryptionkey@0"
I20260927 15:00:58.118371 20428 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:00:58.121409 20442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:58.122970 20428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:00:58.123106 20428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-2/wal
uuid: "a9c49b0da76d4d21a5baa34878ca2a6a"
format_stamp: "Formatted at 2026-09-27 15:00:58 on dist-test-slave-pz0h"
server_key: "89a123d1b669ed1e964b55d677c7b3f4"
server_key_iv: "7bd114fe0d00ce6d162568acda0356bb"
server_key_version: "encryptionkey@0"
I20260927 15:00:58.123238 20428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:58.135892 20428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:58.136595 20428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:58.136739 20428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:58.137292 20428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:00:58.138198 20428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:00:58.138242 20428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:58.138283 20428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:00:58.138302 20428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:58.146751 20428 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44441
I20260927 15:00:58.146823 20555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44441 every 8 connection(s)
I20260927 15:00:58.147763 20428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-2/data/info.pb
I20260927 15:00:58.152204 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 20428
I20260927 15:00:58.152321 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-2/wal/instance
I20260927 15:00:58.152558 392 external_mini_cluster.cc:1428] Setting key a38b09fb9c43c734bc617ffc5ded99de
I20260927 15:00:58.157796 20145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:58.149743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45331 (local address 127.0.98.62:41471)
0927 15:00:58.149947 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:58.149951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:58.151168 (+ 1217us) server_negotiation.cc:407] Connection header received
0927 15:00:58.152672 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:58.152676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:58.152739 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:58.152815 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:58.153949 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:58.154519 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:58.155719 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:58.155949 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:58.156035 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:58.157154 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:58.157158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:58.157161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:00:58.157179 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:58.157204 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:00:58.157214 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:58.157279 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:58.157503 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 15:00:58.157597 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 15:00:58.158946 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:58.150172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41471 (local address 127.0.98.3:45331)
0927 15:00:58.150995 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:58.151043 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:58.152426 (+ 1383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:58.153034 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:58.153044 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:58.153168 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:00:58.153810 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:58.153833 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:58.154629 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:58.154633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:58.155598 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:58.155608 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:58.156091 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:58.156095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:58.156258 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:58.156879 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:00:58.156881 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:00:58.156925 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:00:58.157005 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:00:58.157296 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:00:58.157306 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:00:58.157312 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:00:58.157472 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 15:00:58.157776 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":126,"threads_started":1}
I20260927 15:00:58.160346 20556 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41471
I20260927 15:00:58.160578 20556 heartbeater.cc:464] Registering TS with master...
I20260927 15:00:58.161118 20556 heartbeater.cc:511] Master 127.0.98.62:41471 requested a full tablet report, sending...
I20260927 15:00:58.162207 20106 ts_manager.cc:194] Registered new tserver with Master: a9c49b0da76d4d21a5baa34878ca2a6a (127.0.98.3:44441)
I20260927 15:00:58.162849 20106 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:45331
I20260927 15:00:58.166181 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:00:58.171661 20145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:58.168140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46050 (local address 127.0.98.62:41471)
0927 15:00:58.168292 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:58.168297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:58.168404 (+ 107us) server_negotiation.cc:407] Connection header received
0927 15:00:58.168534 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:58.168538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:58.168580 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:58.168637 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:58.168911 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:58.169423 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:58.170216 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:58.170433 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:58.170516 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:58.171260 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:58.171264 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:58.171266 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:00:58.171280 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:58.171298 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:00:58.171307 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:58.171364 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:58.171457 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 15:00:58.171514 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:00:58.174458 20106 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:00:58.178026 20106 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:00:58.193567 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:58.188261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42643 (local address 127.0.0.1:38328)
0927 15:00:58.188545 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:58.188556 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:58.188657 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:58.189644 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:58.189646 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:58.189662 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:58.189751 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:58.189758 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:58.191149 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:58.191152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:58.192246 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:58.192253 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:58.193049 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:58.193052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:58.193142 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:58.193164 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:00:58.193244 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 15:00:58.193433 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":43,"spinlock_wait_cycles":4224,"thread_start_us":86,"threads_started":1}
I20260927 15:00:58.193558 20572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:58.188548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38328 (local address 127.0.98.2:42643)
0927 15:00:58.189212 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:58.189232 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:58.189255 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:00:58.189327 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:58.189331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:58.189503 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:58.189655 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:58.189875 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:58.191037 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:58.192386 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:58.192927 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:58.193027 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:58.193217 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:58.193272 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 15:00:58.193379 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":116,"threads_started":1}
I20260927 15:00:58.193581 20568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:58.187562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48144 (local address 127.0.98.3:44441)
0927 15:00:58.187987 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:58.187999 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:58.188528 (+ 529us) server_negotiation.cc:407] Connection header received
0927 15:00:58.188841 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:58.188847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:58.189045 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:58.189257 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:58.189822 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:58.190569 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:58.192249 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:58.192855 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:58.192956 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:58.193164 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:58.193300 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 15:00:58.193407 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":87,"threads_started":1}
I20260927 15:00:58.194263 20569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:58.187411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44441 (local address 127.0.0.1:48144)
0927 15:00:58.188377 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:58.188438 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:58.188659 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:58.189456 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:58.189458 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:58.189475 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:58.189671 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:58.189679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:58.190678 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:58.190681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:58.192042 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:58.192053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:58.192978 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:58.192982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:58.193095 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:58.193131 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:00:58.194021 (+ 890us) client_negotiation.cc:240] Negotiation successful
0927 15:00:58.194108 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"mutex_wait_us":8,"spinlock_wait_cycles":4480,"thread_start_us":106,"threads_started":1}
I20260927 15:00:58.195789 20567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:58.187614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41599 (local address 127.0.0.1:37118)
0927 15:00:58.188376 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:58.188438 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:58.188657 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:58.189416 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:58.189423 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:58.189447 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:58.189660 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:58.189668 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:58.192185 (+ 2517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:58.192188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:58.194310 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:58.194322 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:58.195402 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:58.195406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:58.195491 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:58.195514 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:00:58.195571 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 15:00:58.195645 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":1051,"spinlock_wait_cycles":16128,"thread_start_us":131,"threads_started":1}
I20260927 15:00:58.195874 20570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:58.187790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37118 (local address 127.0.98.1:41599)
0927 15:00:58.188194 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:58.188206 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:58.188527 (+ 321us) server_negotiation.cc:407] Connection header received
0927 15:00:58.188832 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:58.188837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:58.189029 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:58.189159 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:58.189797 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:58.191037 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:58.194496 (+ 3459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:58.195269 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:58.195377 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:58.195573 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:58.195623 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 15:00:58.195717 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":105,"threads_started":1}
I20260927 15:00:58.197420 20359 tablet_service.cc:1467] Processing CreateTablet for tablet cdad6e1270194cceadbec5dac85916ae (DEFAULT_TABLE table=test-table [id=5559c9a89f5842c680c1a0765daf1dc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:00:58.198014 20490 tablet_service.cc:1467] Processing CreateTablet for tablet cdad6e1270194cceadbec5dac85916ae (DEFAULT_TABLE table=test-table [id=5559c9a89f5842c680c1a0765daf1dc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:00:58.198997 20359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdad6e1270194cceadbec5dac85916ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:00:58.199072 20490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdad6e1270194cceadbec5dac85916ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:00:58.199339 20228 tablet_service.cc:1467] Processing CreateTablet for tablet cdad6e1270194cceadbec5dac85916ae (DEFAULT_TABLE table=test-table [id=5559c9a89f5842c680c1a0765daf1dc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:00:58.200240 20228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdad6e1270194cceadbec5dac85916ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:00:58.206434 20573 tablet_bootstrap.cc:492] T cdad6e1270194cceadbec5dac85916ae P 94cc1cc8d75343eca1f4fcd9468537e5: Bootstrap starting.
I20260927 15:00:58.207921 20574 tablet_bootstrap.cc:492] T cdad6e1270194cceadbec5dac85916ae P cb17e5fc3480458b824f71c6a47aeaca: Bootstrap starting.
I20260927 15:00:58.209050 20573 tablet_bootstrap.cc:654] T cdad6e1270194cceadbec5dac85916ae P 94cc1cc8d75343eca1f4fcd9468537e5: Neither blocks nor log segments found. Creating new log.
I20260927 15:00:58.209986 20573 log.cc:824] T cdad6e1270194cceadbec5dac85916ae P 94cc1cc8d75343eca1f4fcd9468537e5: Log is configured to *not* fsync() on all Append() calls
I20260927 15:00:58.210206 20574 tablet_bootstrap.cc:654] T cdad6e1270194cceadbec5dac85916ae P cb17e5fc3480458b824f71c6a47aeaca: Neither blocks nor log segments found. Creating new log.
I20260927 15:00:58.210865 20575 tablet_bootstrap.cc:492] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a: Bootstrap starting.
I20260927 15:00:58.211117 20574 log.cc:824] T cdad6e1270194cceadbec5dac85916ae P cb17e5fc3480458b824f71c6a47aeaca: Log is configured to *not* fsync() on all Append() calls
I20260927 15:00:58.212416 20573 tablet_bootstrap.cc:492] T cdad6e1270194cceadbec5dac85916ae P 94cc1cc8d75343eca1f4fcd9468537e5: No bootstrap required, opened a new log
I20260927 15:00:58.212630 20573 ts_tablet_manager.cc:1397] T cdad6e1270194cceadbec5dac85916ae P 94cc1cc8d75343eca1f4fcd9468537e5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:00:58.213176 20575 tablet_bootstrap.cc:654] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a: Neither blocks nor log segments found. Creating new log.
I20260927 15:00:58.213229 20574 tablet_bootstrap.cc:492] T cdad6e1270194cceadbec5dac85916ae P cb17e5fc3480458b824f71c6a47aeaca: No bootstrap required, opened a new log
I20260927 15:00:58.213430 20574 ts_tablet_manager.cc:1397] T cdad6e1270194cceadbec5dac85916ae P cb17e5fc3480458b824f71c6a47aeaca: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:00:58.214099 20575 log.cc:824] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a: Log is configured to *not* fsync() on all Append() calls
I20260927 15:00:58.216372 20575 tablet_bootstrap.cc:492] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a: No bootstrap required, opened a new log
I20260927 15:00:58.216598 20575 ts_tablet_manager.cc:1397] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:00:58.216542 20573 raft_consensus.cc:348] T cdad6e1270194cceadbec5dac85916ae P 94cc1cc8d75343eca1f4fcd9468537e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c49b0da76d4d21a5baa34878ca2a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44441 } } peers { permanent_uuid: "cb17e5fc3480458b824f71c6a47aeaca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42643 } } peers { permanent_uuid: "94cc1cc8d75343eca1f4fcd9468537e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41599 } }
I20260927 15:00:58.216840 20573 raft_consensus.cc:374] T cdad6e1270194cceadbec5dac85916ae P 94cc1cc8d75343eca1f4fcd9468537e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:00:58.216894 20573 raft_consensus.cc:729] T cdad6e1270194cceadbec5dac85916ae P 94cc1cc8d75343eca1f4fcd9468537e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94cc1cc8d75343eca1f4fcd9468537e5, State: Initialized, Role: FOLLOWER
I20260927 15:00:58.217403 20573 consensus_queue.cc:260] T cdad6e1270194cceadbec5dac85916ae P 94cc1cc8d75343eca1f4fcd9468537e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c49b0da76d4d21a5baa34878ca2a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44441 } } peers { permanent_uuid: "cb17e5fc3480458b824f71c6a47aeaca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42643 } } peers { permanent_uuid: "94cc1cc8d75343eca1f4fcd9468537e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41599 } }
I20260927 15:00:58.218073 20294 heartbeater.cc:503] Master 127.0.98.62:41471 was elected leader, sending a full tablet report...
I20260927 15:00:58.218047 20574 raft_consensus.cc:348] T cdad6e1270194cceadbec5dac85916ae P cb17e5fc3480458b824f71c6a47aeaca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c49b0da76d4d21a5baa34878ca2a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44441 } } peers { permanent_uuid: "cb17e5fc3480458b824f71c6a47aeaca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42643 } } peers { permanent_uuid: "94cc1cc8d75343eca1f4fcd9468537e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41599 } }
I20260927 15:00:58.218386 20574 raft_consensus.cc:374] T cdad6e1270194cceadbec5dac85916ae P cb17e5fc3480458b824f71c6a47aeaca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:00:58.218436 20574 raft_consensus.cc:729] T cdad6e1270194cceadbec5dac85916ae P cb17e5fc3480458b824f71c6a47aeaca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb17e5fc3480458b824f71c6a47aeaca, State: Initialized, Role: FOLLOWER
I20260927 15:00:58.218528 20573 ts_tablet_manager.cc:1428] T cdad6e1270194cceadbec5dac85916ae P 94cc1cc8d75343eca1f4fcd9468537e5: Time spent starting tablet: real 0.006s user 0.001s sys 0.006s
I20260927 15:00:58.218997 20574 consensus_queue.cc:260] T cdad6e1270194cceadbec5dac85916ae P cb17e5fc3480458b824f71c6a47aeaca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c49b0da76d4d21a5baa34878ca2a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44441 } } peers { permanent_uuid: "cb17e5fc3480458b824f71c6a47aeaca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42643 } } peers { permanent_uuid: "94cc1cc8d75343eca1f4fcd9468537e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41599 } }
I20260927 15:00:58.219754 20425 heartbeater.cc:503] Master 127.0.98.62:41471 was elected leader, sending a full tablet report...
I20260927 15:00:58.220052 20574 ts_tablet_manager.cc:1428] T cdad6e1270194cceadbec5dac85916ae P cb17e5fc3480458b824f71c6a47aeaca: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260927 15:00:58.220845 20575 raft_consensus.cc:348] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c49b0da76d4d21a5baa34878ca2a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44441 } } peers { permanent_uuid: "cb17e5fc3480458b824f71c6a47aeaca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42643 } } peers { permanent_uuid: "94cc1cc8d75343eca1f4fcd9468537e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41599 } }
I20260927 15:00:58.221084 20575 raft_consensus.cc:374] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:00:58.221134 20575 raft_consensus.cc:729] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9c49b0da76d4d21a5baa34878ca2a6a, State: Initialized, Role: FOLLOWER
I20260927 15:00:58.221626 20575 consensus_queue.cc:260] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c49b0da76d4d21a5baa34878ca2a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44441 } } peers { permanent_uuid: "cb17e5fc3480458b824f71c6a47aeaca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42643 } } peers { permanent_uuid: "94cc1cc8d75343eca1f4fcd9468537e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41599 } }
I20260927 15:00:58.222645 20575 ts_tablet_manager.cc:1428] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 15:00:58.223023 20556 heartbeater.cc:503] Master 127.0.98.62:41471 was elected leader, sending a full tablet report...
I20260927 15:00:58.240168 20581 raft_consensus.cc:482] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:00:58.240418 20581 raft_consensus.cc:504] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c49b0da76d4d21a5baa34878ca2a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44441 } } peers { permanent_uuid: "cb17e5fc3480458b824f71c6a47aeaca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42643 } } peers { permanent_uuid: "94cc1cc8d75343eca1f4fcd9468537e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41599 } }
I20260927 15:00:58.241724 20581 leader_election.cc:288] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb17e5fc3480458b824f71c6a47aeaca (127.0.98.2:42643), 94cc1cc8d75343eca1f4fcd9468537e5 (127.0.98.1:41599)
I20260927 15:00:58.245520 20570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:58.242063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58187 (local address 127.0.98.1:41599)
0927 15:00:58.242171 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:58.242175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:58.242189 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:00:58.242256 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:58.242259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:58.242311 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:58.242394 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:58.242676 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:58.243309 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:58.244417 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:58.244984 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:58.245080 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:58.245150 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:58.245350 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 15:00:58.245408 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:00:58.245543 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:58.241943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41599 (local address 127.0.98.3:58187)
0927 15:00:58.242049 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:58.242064 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:58.242167 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:58.242412 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:58.242415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:58.242433 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:58.242539 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:58.242546 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:58.243410 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:58.243414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:58.244270 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:58.244280 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:58.245108 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:58.245112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:58.245217 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:58.245256 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 15:00:58.245332 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 15:00:58.245408 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 15:00:58.246198 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:58.241943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42643 (local address 127.0.98.3:52869)
0927 15:00:58.242422 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:58.242436 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:00:58.242532 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:00:58.242736 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:00:58.242738 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:00:58.242755 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:00:58.242897 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:58.242904 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:58.243721 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:58.243726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:58.244940 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:00:58.244951 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:00:58.245809 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:00:58.245813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:00:58.245910 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:58.245935 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 15:00:58.246003 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 15:00:58.246060 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":120,"thread_start_us":133,"threads_started":1}
I20260927 15:00:58.246232 20572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:58.242048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52869 (local address 127.0.98.2:42643)
0927 15:00:58.242164 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:58.242169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:58.242521 (+ 352us) server_negotiation.cc:407] Connection header received
0927 15:00:58.242607 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:58.242610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:58.242660 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:58.242714 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:00:58.242994 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:58.243512 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:58.245089 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:58.245651 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:58.245778 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:58.245836 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:58.246041 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 15:00:58.246119 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:00:58.246297 20248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdad6e1270194cceadbec5dac85916ae" candidate_uuid: "a9c49b0da76d4d21a5baa34878ca2a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94cc1cc8d75343eca1f4fcd9468537e5" is_pre_election: true
I20260927 15:00:58.246649 20248 raft_consensus.cc:2454] T cdad6e1270194cceadbec5dac85916ae P 94cc1cc8d75343eca1f4fcd9468537e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9c49b0da76d4d21a5baa34878ca2a6a in term 0.
I20260927 15:00:58.246840 20379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdad6e1270194cceadbec5dac85916ae" candidate_uuid: "a9c49b0da76d4d21a5baa34878ca2a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb17e5fc3480458b824f71c6a47aeaca" is_pre_election: true
I20260927 15:00:58.247175 20379 raft_consensus.cc:2454] T cdad6e1270194cceadbec5dac85916ae P cb17e5fc3480458b824f71c6a47aeaca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9c49b0da76d4d21a5baa34878ca2a6a in term 0.
I20260927 15:00:58.247242 20443 leader_election.cc:302] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94cc1cc8d75343eca1f4fcd9468537e5, a9c49b0da76d4d21a5baa34878ca2a6a; no voters:
I20260927 15:00:58.247756 20581 raft_consensus.cc:2781] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:00:58.247828 20581 raft_consensus.cc:482] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:00:58.247855 20581 raft_consensus.cc:3037] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:00:58.248837 20581 raft_consensus.cc:504] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c49b0da76d4d21a5baa34878ca2a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44441 } } peers { permanent_uuid: "cb17e5fc3480458b824f71c6a47aeaca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42643 } } peers { permanent_uuid: "94cc1cc8d75343eca1f4fcd9468537e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41599 } }
I20260927 15:00:58.249173 20581 leader_election.cc:288] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a [CANDIDATE]: Term 1 election: Requested vote from peers cb17e5fc3480458b824f71c6a47aeaca (127.0.98.2:42643), 94cc1cc8d75343eca1f4fcd9468537e5 (127.0.98.1:41599)
I20260927 15:00:58.249571 20379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdad6e1270194cceadbec5dac85916ae" candidate_uuid: "a9c49b0da76d4d21a5baa34878ca2a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb17e5fc3480458b824f71c6a47aeaca"
I20260927 15:00:58.249571 20248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdad6e1270194cceadbec5dac85916ae" candidate_uuid: "a9c49b0da76d4d21a5baa34878ca2a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94cc1cc8d75343eca1f4fcd9468537e5"
I20260927 15:00:58.249714 20379 raft_consensus.cc:3037] T cdad6e1270194cceadbec5dac85916ae P cb17e5fc3480458b824f71c6a47aeaca [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:00:58.249714 20248 raft_consensus.cc:3037] T cdad6e1270194cceadbec5dac85916ae P 94cc1cc8d75343eca1f4fcd9468537e5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:00:58.251026 20248 raft_consensus.cc:2454] T cdad6e1270194cceadbec5dac85916ae P 94cc1cc8d75343eca1f4fcd9468537e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9c49b0da76d4d21a5baa34878ca2a6a in term 1.
I20260927 15:00:58.251026 20379 raft_consensus.cc:2454] T cdad6e1270194cceadbec5dac85916ae P cb17e5fc3480458b824f71c6a47aeaca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9c49b0da76d4d21a5baa34878ca2a6a in term 1.
I20260927 15:00:58.251367 20443 leader_election.cc:302] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94cc1cc8d75343eca1f4fcd9468537e5, a9c49b0da76d4d21a5baa34878ca2a6a; no voters:
I20260927 15:00:58.251585 20581 raft_consensus.cc:2781] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:00:58.251850 20581 raft_consensus.cc:686] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a [term 1 LEADER]: Becoming Leader. State: Replica: a9c49b0da76d4d21a5baa34878ca2a6a, State: Running, Role: LEADER
I20260927 15:00:58.252192 20581 consensus_queue.cc:237] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c49b0da76d4d21a5baa34878ca2a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44441 } } peers { permanent_uuid: "cb17e5fc3480458b824f71c6a47aeaca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42643 } } peers { permanent_uuid: "94cc1cc8d75343eca1f4fcd9468537e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41599 } }
W20260927 15:00:58.254637 20295 tablet.cc:2369] T cdad6e1270194cceadbec5dac85916ae P 94cc1cc8d75343eca1f4fcd9468537e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:00:58.255561 20106 catalog_manager.cc:5495] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a reported cstate change: term changed from 0 to 1, leader changed from <none> to a9c49b0da76d4d21a5baa34878ca2a6a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a9c49b0da76d4d21a5baa34878ca2a6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c49b0da76d4d21a5baa34878ca2a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb17e5fc3480458b824f71c6a47aeaca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94cc1cc8d75343eca1f4fcd9468537e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41599 } health_report { overall_health: UNKNOWN } } }
I20260927 15:00:58.279148 20568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:58.275291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48158 (local address 127.0.98.3:44441)
0927 15:00:58.275415 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:58.275419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:58.275432 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:00:58.275528 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:58.275530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:58.275584 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:58.275670 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:00:58.275886 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:58.276391 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:58.277288 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:58.277506 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:58.277594 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:58.278195 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:00:58.278657 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:00:58.278765 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:58.278932 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 15:00:58.278989 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:00:58.285501 20379 raft_consensus.cc:1261] T cdad6e1270194cceadbec5dac85916ae P cb17e5fc3480458b824f71c6a47aeaca [term 1 FOLLOWER]: Refusing update from remote peer a9c49b0da76d4d21a5baa34878ca2a6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:00:58.285501 20248 raft_consensus.cc:1261] T cdad6e1270194cceadbec5dac85916ae P 94cc1cc8d75343eca1f4fcd9468537e5 [term 1 FOLLOWER]: Refusing update from remote peer a9c49b0da76d4d21a5baa34878ca2a6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:00:58.286278 20583 consensus_queue.cc:1035] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "94cc1cc8d75343eca1f4fcd9468537e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:00:58.286463 20581 consensus_queue.cc:1035] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb17e5fc3480458b824f71c6a47aeaca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:00:58.294392 20586 mvcc.cc:205] Tried to move back new op lower bound from 7333975073930407936 to 7333975073803849728. Current Snapshot: MvccSnapshot[applied={T|T < 7333975073930407936}]
I20260927 15:00:58.297241 20587 mvcc.cc:205] Tried to move back new op lower bound from 7333975073930407936 to 7333975073803849728. Current Snapshot: MvccSnapshot[applied={T|T < 7333975073930407936}]
I20260927 15:00:58.303857 20106 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:00:58.304040 20106 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:00:58.306562 20106 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b128e5107ec64d808f896b1a16fc5933: Sending DeleteTablet for 3 replicas of tablet cdad6e1270194cceadbec5dac85916ae
I20260927 15:00:58.307574 20228 tablet_service.cc:1514] Processing DeleteTablet for tablet cdad6e1270194cceadbec5dac85916ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:00:58 UTC) from {username='slave'} at 127.0.0.1:37118
I20260927 15:00:58.307721 20359 tablet_service.cc:1514] Processing DeleteTablet for tablet cdad6e1270194cceadbec5dac85916ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:00:58 UTC) from {username='slave'} at 127.0.0.1:38328
I20260927 15:00:58.308161 20596 tablet_replica.cc:330] T cdad6e1270194cceadbec5dac85916ae P 94cc1cc8d75343eca1f4fcd9468537e5: stopping tablet replica
I20260927 15:00:58.308156 20490 tablet_service.cc:1514] Processing DeleteTablet for tablet cdad6e1270194cceadbec5dac85916ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:00:58 UTC) from {username='slave'} at 127.0.0.1:48144
I20260927 15:00:58.308562 20596 raft_consensus.cc:2229] T cdad6e1270194cceadbec5dac85916ae P 94cc1cc8d75343eca1f4fcd9468537e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:00:58.308604 20598 tablet_replica.cc:330] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a: stopping tablet replica
I20260927 15:00:58.308712 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 20166
I20260927 15:00:58.308939 20596 raft_consensus.cc:2258] T cdad6e1270194cceadbec5dac85916ae P 94cc1cc8d75343eca1f4fcd9468537e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:00:58.309312 20597 tablet_replica.cc:330] T cdad6e1270194cceadbec5dac85916ae P cb17e5fc3480458b824f71c6a47aeaca: stopping tablet replica
I20260927 15:00:58.309624 20597 raft_consensus.cc:2229] T cdad6e1270194cceadbec5dac85916ae P cb17e5fc3480458b824f71c6a47aeaca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:00:58.309902 20597 raft_consensus.cc:2258] T cdad6e1270194cceadbec5dac85916ae P cb17e5fc3480458b824f71c6a47aeaca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:00:58.310521 20598 raft_consensus.cc:2229] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:00:58.310969 20598 raft_consensus.cc:2258] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:00:58.311987 20597 ts_tablet_manager.cc:1905] T cdad6e1270194cceadbec5dac85916ae P cb17e5fc3480458b824f71c6a47aeaca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:00:58.312371 20598 ts_tablet_manager.cc:1905] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:00:58.314859 20597 ts_tablet_manager.cc:1918] T cdad6e1270194cceadbec5dac85916ae P cb17e5fc3480458b824f71c6a47aeaca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:00:58.314975 20597 log.cc:1196] T cdad6e1270194cceadbec5dac85916ae P cb17e5fc3480458b824f71c6a47aeaca: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-1/wal/wals/cdad6e1270194cceadbec5dac85916ae
I20260927 15:00:58.315146 20598 ts_tablet_manager.cc:1918] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:00:58.315275 20597 ts_tablet_manager.cc:1939] T cdad6e1270194cceadbec5dac85916ae P cb17e5fc3480458b824f71c6a47aeaca: Deleting consensus metadata
I20260927 15:00:58.315371 20598 log.cc:1196] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a: Deleting WAL directory at /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790521004237705-392-0/minicluster-data/ts-2/wal/wals/cdad6e1270194cceadbec5dac85916ae
I20260927 15:00:58.315706 20598 ts_tablet_manager.cc:1939] T cdad6e1270194cceadbec5dac85916ae P a9c49b0da76d4d21a5baa34878ca2a6a: Deleting consensus metadata
I20260927 15:00:58.316718 20092 catalog_manager.cc:4841] TS cb17e5fc3480458b824f71c6a47aeaca (127.0.98.2:42643): tablet cdad6e1270194cceadbec5dac85916ae (table test-table [id=5559c9a89f5842c680c1a0765daf1dc3]) successfully deleted
I20260927 15:00:58.316947 20092 catalog_manager.cc:4841] TS a9c49b0da76d4d21a5baa34878ca2a6a (127.0.98.3:44441): tablet cdad6e1270194cceadbec5dac85916ae (table test-table [id=5559c9a89f5842c680c1a0765daf1dc3]) successfully deleted
W20260927 15:00:58.317160 20090 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41599 (error 108)
I20260927 15:00:58.318037 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 20297
I20260927 15:00:58.318202 20567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:58.317909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41599 (local address 127.0.0.1:37134)
0927 15:00:58.318043 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 15:00:58.318108 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 15:00:58.319470 20090 catalog_manager.cc:4584] TS 94cc1cc8d75343eca1f4fcd9468537e5 (127.0.98.1:41599): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cdad6e1270194cceadbec5dac85916ae: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41599: connect: Connection refused (error 111)
I20260927 15:00:58.323843 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 20428
I20260927 15:00:58.329407 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 20075
2026-09-27T15:00:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1061 ms)
[----------] 6 tests from EncryptionPolicyTest (10932 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:00:58 dist-test-slave-pz0h krb5kdc[20602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:00:58 dist-test-slave-pz0h krb5kdc[20602](info): set up 2 sockets
Sep 27 15:00:58 dist-test-slave-pz0h krb5kdc[20602](info): commencing operation
krb5kdc: starting...
W20260927 15:00:59.282685 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 15:00:59 dist-test-slave-pz0h krb5kdc[20602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:00:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:00:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:00:59.401650 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40437
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:00:59.513499 20615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:00:59.513799 20615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:00:59.513841 20615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:00:59.518071 20615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:00:59.518160 20615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:00:59.518179 20615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:00:59.518196 20615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:00:59.518210 20615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:00:59.524001 20615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40437
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:00:59.525122 20615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:00:59.526145 20615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:00:59.534435 20620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:00:59.534420 20623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:59.534484 20615 server_base.cc:1034] running on GCE node
W20260927 15:00:59.534433 20621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:00:59.535234 20615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:00:59.536309 20615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:00:59.537461 20615 hybrid_clock.cc:648] HybridClock initialized: now 1790521259537446 us; error 29 us; skew 500 ppm
Sep 27 15:00:59 dist-test-slave-pz0h krb5kdc[20602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:00:59.540313 20615 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:00:59.541329 20615 webserver.cc:460] Webserver started at http://127.0.0.1:45367/ using document root <none> and password file <none>
I20260927 15:00:59.541908 20615 fs_manager.cc:359] Metadata directory not provided
I20260927 15:00:59.541970 20615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:00:59.542196 20615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:00:59.544440 20615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "f95b4adfda6e4afdaec55cb8c300638f"
format_stamp: "Formatted at 2026-09-27 15:00:59 on dist-test-slave-pz0h"
server_key: "f0287ff9661f04c5b4874918b16bae64"
server_key_iv: "4c70684d83062dd12afb9b2b83bcd329"
server_key_version: "encryptionkey@0"
I20260927 15:00:59.545020 20615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "f95b4adfda6e4afdaec55cb8c300638f"
format_stamp: "Formatted at 2026-09-27 15:00:59 on dist-test-slave-pz0h"
server_key: "f0287ff9661f04c5b4874918b16bae64"
server_key_iv: "4c70684d83062dd12afb9b2b83bcd329"
server_key_version: "encryptionkey@0"
I20260927 15:00:59.548909 20615 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:00:59.551509 20630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:00:59.552788 20615 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:00:59.552922 20615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "f95b4adfda6e4afdaec55cb8c300638f"
format_stamp: "Formatted at 2026-09-27 15:00:59 on dist-test-slave-pz0h"
server_key: "f0287ff9661f04c5b4874918b16bae64"
server_key_iv: "4c70684d83062dd12afb9b2b83bcd329"
server_key_version: "encryptionkey@0"
I20260927 15:00:59.553046 20615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:00:59.576022 20615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:00:59.579257 20615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:00:59.579438 20615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:00:59.586321 20615 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40437
I20260927 15:00:59.586357 20682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40437 every 8 connection(s)
I20260927 15:00:59.587332 20615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:00:59.588258 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 20615
I20260927 15:00:59.588388 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 15:00:59.588670 392 external_mini_cluster.cc:1428] Setting key da0255d34c352eef9ead63329b41844e
I20260927 15:00:59.590730 20683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:00:59.596659 20683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f95b4adfda6e4afdaec55cb8c300638f: Bootstrap starting.
Sep 27 15:00:59 dist-test-slave-pz0h krb5kdc[20602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 15:00:59.599293 20683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f95b4adfda6e4afdaec55cb8c300638f: Neither blocks nor log segments found. Creating new log.
I20260927 15:00:59.600121 20683 log.cc:824] T 00000000000000000000000000000000 P f95b4adfda6e4afdaec55cb8c300638f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:00:59.602264 20683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f95b4adfda6e4afdaec55cb8c300638f: No bootstrap required, opened a new log
I20260927 15:00:59.605130 20686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:59.590137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40619 (local address 127.0.0.1:40437)
0927 15:00:59.590714 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:59.590735 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:59.590774 (+ 39us) server_negotiation.cc:407] Connection header received
0927 15:00:59.591486 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:59.591513 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:59.591882 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:59.592215 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:59.593222 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:59.595261 (+ 2039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:59.596369 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:59.596927 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:59.597094 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:59.600009 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:59.600017 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:59.600034 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:00:59.600069 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:59.601704 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:59.602241 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:59.602246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:59.602251 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:59.602395 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:59.602837 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:59.602840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:59.602843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:59.603273 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:59.603443 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:59.604101 (+ 658us) server_negotiation.cc:299] Negotiation successful
0927 15:00:59.604354 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":141,"threads_started":1}
I20260927 15:00:59.606964 20683 raft_consensus.cc:348] T 00000000000000000000000000000000 P f95b4adfda6e4afdaec55cb8c300638f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95b4adfda6e4afdaec55cb8c300638f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40437 } }
I20260927 15:00:59.607235 20683 raft_consensus.cc:374] T 00000000000000000000000000000000 P f95b4adfda6e4afdaec55cb8c300638f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:00:59.607298 20683 raft_consensus.cc:729] T 00000000000000000000000000000000 P f95b4adfda6e4afdaec55cb8c300638f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f95b4adfda6e4afdaec55cb8c300638f, State: Initialized, Role: FOLLOWER
I20260927 15:00:59.607764 20683 consensus_queue.cc:260] T 00000000000000000000000000000000 P f95b4adfda6e4afdaec55cb8c300638f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95b4adfda6e4afdaec55cb8c300638f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40437 } }
I20260927 15:00:59.607924 20683 raft_consensus.cc:388] T 00000000000000000000000000000000 P f95b4adfda6e4afdaec55cb8c300638f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:00:59.608000 20683 raft_consensus.cc:482] T 00000000000000000000000000000000 P f95b4adfda6e4afdaec55cb8c300638f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:00:59.608096 20683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f95b4adfda6e4afdaec55cb8c300638f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:00:59.608446 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:00:59.609148 20683 raft_consensus.cc:504] T 00000000000000000000000000000000 P f95b4adfda6e4afdaec55cb8c300638f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95b4adfda6e4afdaec55cb8c300638f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40437 } }
I20260927 15:00:59.609495 20683 leader_election.cc:302] T 00000000000000000000000000000000 P f95b4adfda6e4afdaec55cb8c300638f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f95b4adfda6e4afdaec55cb8c300638f; no voters:
I20260927 15:00:59.609848 20683 leader_election.cc:288] T 00000000000000000000000000000000 P f95b4adfda6e4afdaec55cb8c300638f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:00:59.609947 20692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f95b4adfda6e4afdaec55cb8c300638f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:00:59.610152 20692 raft_consensus.cc:686] T 00000000000000000000000000000000 P f95b4adfda6e4afdaec55cb8c300638f [term 1 LEADER]: Becoming Leader. State: Replica: f95b4adfda6e4afdaec55cb8c300638f, State: Running, Role: LEADER
I20260927 15:00:59.611049 20692 consensus_queue.cc:237] T 00000000000000000000000000000000 P f95b4adfda6e4afdaec55cb8c300638f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95b4adfda6e4afdaec55cb8c300638f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40437 } }
I20260927 15:00:59.611254 20683 sys_catalog.cc:564] T 00000000000000000000000000000000 P f95b4adfda6e4afdaec55cb8c300638f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:00:59.612599 20693 sys_catalog.cc:455] T 00000000000000000000000000000000 P f95b4adfda6e4afdaec55cb8c300638f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f95b4adfda6e4afdaec55cb8c300638f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95b4adfda6e4afdaec55cb8c300638f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40437 } } }
I20260927 15:00:59.612725 20693 sys_catalog.cc:458] T 00000000000000000000000000000000 P f95b4adfda6e4afdaec55cb8c300638f [sys.catalog]: This master's current role is: LEADER
I20260927 15:00:59.613238 20703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:00:59.613173 20694 sys_catalog.cc:455] T 00000000000000000000000000000000 P f95b4adfda6e4afdaec55cb8c300638f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f95b4adfda6e4afdaec55cb8c300638f. Latest consensus state: current_term: 1 leader_uuid: "f95b4adfda6e4afdaec55cb8c300638f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95b4adfda6e4afdaec55cb8c300638f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40437 } } }
I20260927 15:00:59.613516 20694 sys_catalog.cc:458] T 00000000000000000000000000000000 P f95b4adfda6e4afdaec55cb8c300638f [sys.catalog]: This master's current role is: LEADER
I20260927 15:00:59.617409 20703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:00:59.619064 20686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:00:59.610546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50043 (local address 127.0.0.1:40437)
0927 15:00:59.610678 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 15:00:59.610683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:00:59.610953 (+ 270us) server_negotiation.cc:407] Connection header received
0927 15:00:59.611087 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:00:59.611090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:00:59.611142 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:00:59.611223 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:00:59.612169 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:59.612945 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:59.614009 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:00:59.614246 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:00:59.614333 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:00:59.615944 (+ 1611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:00:59.615948 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:00:59.615951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:00:59.615991 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:00:59.617350 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:59.617834 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:59.617838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:59.617840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:59.617915 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:00:59.618219 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:00:59.618223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:00:59.618224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:00:59.618398 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:00:59.618540 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 15:00:59.618814 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 15:00:59.618908 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:00:59.623651 20703 catalog_manager.cc:1347] Generated new cluster ID: eecb007a2ed14279830cd91730fbbed1
I20260927 15:00:59.623737 20703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:00:59.630131 20703 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:00:59.631520 20703 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:00:59.640017 20703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f95b4adfda6e4afdaec55cb8c300638f: Generated new TSK 0
I20260927 15:00:59.640990 20703 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:00:59.644532 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 20615
W20260927 15:00:59.649808 20600 connection.cc:535] client connection to 127.0.0.1:40437 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40437): Connection reset by peer (error 104)
2026-09-27T15:00:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:00:59 dist-test-slave-pz0h krb5kdc[20714](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:00:59 dist-test-slave-pz0h krb5kdc[20714](info): set up 2 sockets
Sep 27 15:00:59 dist-test-slave-pz0h krb5kdc[20714](info): commencing operation
krb5kdc: starting...
W20260927 15:01:00.602658 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 15:01:00 dist-test-slave-pz0h krb5kdc[20714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:01:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:01:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:01:00.725809 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44483
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:01:00.840376 20727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:01:00.840687 20727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:01:00.840747 20727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:01:00.845005 20727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:01:00.845098 20727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:01:00.845125 20727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:01:00.845148 20727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:01:00.845177 20727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:01:00.850992 20727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44483
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:01:00.852180 20727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:01:00.853252 20727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:01:00.861905 20733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:00.861909 20732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:00.861905 20735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:01:00.862174 20727 server_base.cc:1034] running on GCE node
I20260927 15:01:00.862886 20727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:01:00.864087 20727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:01:00.865286 20727 hybrid_clock.cc:648] HybridClock initialized: now 1790521260865262 us; error 40 us; skew 500 ppm
Sep 27 15:01:00 dist-test-slave-pz0h krb5kdc[20714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:01:00.868258 20727 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:01:00.869282 20727 webserver.cc:460] Webserver started at http://127.0.0.1:45825/ using document root <none> and password file <none>
I20260927 15:01:00.869858 20727 fs_manager.cc:359] Metadata directory not provided
I20260927 15:01:00.869931 20727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:01:00.870155 20727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:01:00.872377 20727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "451907b0108947d4bda9eca16c0a76df"
format_stamp: "Formatted at 2026-09-27 15:01:00 on dist-test-slave-pz0h"
server_key: "6789e1cc41eececbd3b9ef40322f11a2"
server_key_iv: "85c9d78052651285ab60cf0a8eb37fab"
server_key_version: "encryptionkey@0"
I20260927 15:01:00.872996 20727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "451907b0108947d4bda9eca16c0a76df"
format_stamp: "Formatted at 2026-09-27 15:01:00 on dist-test-slave-pz0h"
server_key: "6789e1cc41eececbd3b9ef40322f11a2"
server_key_iv: "85c9d78052651285ab60cf0a8eb37fab"
server_key_version: "encryptionkey@0"
I20260927 15:01:00.877146 20727 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:01:00.879783 20742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:01:00.881135 20727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:01:00.881271 20727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "451907b0108947d4bda9eca16c0a76df"
format_stamp: "Formatted at 2026-09-27 15:01:00 on dist-test-slave-pz0h"
server_key: "6789e1cc41eececbd3b9ef40322f11a2"
server_key_iv: "85c9d78052651285ab60cf0a8eb37fab"
server_key_version: "encryptionkey@0"
I20260927 15:01:00.881390 20727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:01:00.896781 20727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:01:00.900038 20727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:01:00.900231 20727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:01:00.907021 20727 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44483
I20260927 15:01:00.907034 20794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44483 every 8 connection(s)
I20260927 15:01:00.908010 20727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:01:00.910902 20795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:01:00.912372 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 20727
I20260927 15:01:00.912514 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 15:01:00.912762 392 external_mini_cluster.cc:1428] Setting key 4da3cbe66bc4e4e1f993c56a18053b88
I20260927 15:01:00.917925 20795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 451907b0108947d4bda9eca16c0a76df: Bootstrap starting.
I20260927 15:01:00.920778 20795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 451907b0108947d4bda9eca16c0a76df: Neither blocks nor log segments found. Creating new log.
I20260927 15:01:00.921586 20795 log.cc:824] T 00000000000000000000000000000000 P 451907b0108947d4bda9eca16c0a76df: Log is configured to *not* fsync() on all Append() calls
I20260927 15:01:00.923072 20798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:00.914199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38285 (local address 127.0.0.1:44483)
0927 15:01:00.914654 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:00.914674 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:00.914720 (+ 46us) server_negotiation.cc:407] Connection header received
0927 15:01:00.915598 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:00.915624 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:00.916077 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:00.916472 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:00.916910 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:00.918352 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:00.919322 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:00.920036 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:00.920215 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:00.921077 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:00.921083 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:00.921098 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:01:00.921535 (+ 437us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:00.921599 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:01:00.921624 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:00.921759 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:00.921949 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 15:01:00.922102 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":118,"threads_started":1}
I20260927 15:01:00.923935 20795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 451907b0108947d4bda9eca16c0a76df: No bootstrap required, opened a new log
I20260927 15:01:00.925937 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:01:00.927853 20795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 451907b0108947d4bda9eca16c0a76df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451907b0108947d4bda9eca16c0a76df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44483 } }
I20260927 15:01:00.928156 20795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 451907b0108947d4bda9eca16c0a76df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:01:00.928231 20795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 451907b0108947d4bda9eca16c0a76df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451907b0108947d4bda9eca16c0a76df, State: Initialized, Role: FOLLOWER
I20260927 15:01:00.928702 20795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 451907b0108947d4bda9eca16c0a76df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451907b0108947d4bda9eca16c0a76df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44483 } }
I20260927 15:01:00.928865 20795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 451907b0108947d4bda9eca16c0a76df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:01:00.928930 20795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 451907b0108947d4bda9eca16c0a76df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:01:00.929039 20795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 451907b0108947d4bda9eca16c0a76df [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:01:00.930123 20795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 451907b0108947d4bda9eca16c0a76df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451907b0108947d4bda9eca16c0a76df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44483 } }
I20260927 15:01:00.930503 20795 leader_election.cc:302] T 00000000000000000000000000000000 P 451907b0108947d4bda9eca16c0a76df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 451907b0108947d4bda9eca16c0a76df; no voters:
I20260927 15:01:00.930913 20795 leader_election.cc:288] T 00000000000000000000000000000000 P 451907b0108947d4bda9eca16c0a76df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:01:00.931025 20805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 451907b0108947d4bda9eca16c0a76df [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:01:00.931246 20805 raft_consensus.cc:686] T 00000000000000000000000000000000 P 451907b0108947d4bda9eca16c0a76df [term 1 LEADER]: Becoming Leader. State: Replica: 451907b0108947d4bda9eca16c0a76df, State: Running, Role: LEADER
I20260927 15:01:00.931581 20805 consensus_queue.cc:237] T 00000000000000000000000000000000 P 451907b0108947d4bda9eca16c0a76df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451907b0108947d4bda9eca16c0a76df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44483 } }
I20260927 15:01:00.932068 20798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:00.927874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50617 (local address 127.0.0.1:44483)
0927 15:01:00.927996 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:00.928001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:00.928129 (+ 128us) server_negotiation.cc:407] Connection header received
0927 15:01:00.928279 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:00.928283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:00.928335 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:00.928415 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:00.928702 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:00.929424 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:00.930232 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:00.930557 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:00.930658 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:00.931559 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:00.931563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:00.931565 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:01:00.931580 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:00.931610 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:01:00.931623 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:00.931706 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:00.931894 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 15:01:00.931936 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:01:00.932328 20795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 451907b0108947d4bda9eca16c0a76df [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:01:00.933290 20806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 451907b0108947d4bda9eca16c0a76df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "451907b0108947d4bda9eca16c0a76df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451907b0108947d4bda9eca16c0a76df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44483 } } }
I20260927 15:01:00.933418 20806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 451907b0108947d4bda9eca16c0a76df [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:00.933693 20807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 451907b0108947d4bda9eca16c0a76df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 451907b0108947d4bda9eca16c0a76df. Latest consensus state: current_term: 1 leader_uuid: "451907b0108947d4bda9eca16c0a76df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451907b0108947d4bda9eca16c0a76df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44483 } } }
I20260927 15:01:00.933784 20807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 451907b0108947d4bda9eca16c0a76df [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:00.933979 20814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:01:00.936764 20814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:01:00.942495 20814 catalog_manager.cc:1347] Generated new cluster ID: a6207117b787432b9e621941e87b4ac4
I20260927 15:01:00.942580 20814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:01:00.951619 20814 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:01:00.952780 20814 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:01:00.958252 20814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 451907b0108947d4bda9eca16c0a76df: Generated new TSK 0
I20260927 15:01:00.959306 20814 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:01:00.964915 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 20727
W20260927 15:01:00.969842 20712 connection.cc:535] client connection to 127.0.0.1:44483 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44483): Connection reset by peer (error 104)
2026-09-27T15:01:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:01:01 dist-test-slave-pz0h krb5kdc[20826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:01:01 dist-test-slave-pz0h krb5kdc[20826](info): set up 2 sockets
Sep 27 15:01:01 dist-test-slave-pz0h krb5kdc[20826](info): commencing operation
krb5kdc: starting...
W20260927 15:01:01.918715 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 15:01:01 dist-test-slave-pz0h krb5kdc[20826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:01:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:01:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:01:02.041777 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36195
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:01:02.163451 20839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:01:02.163759 20839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:01:02.163805 20839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:01:02.168032 20839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:01:02.168109 20839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:01:02.168128 20839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:01:02.168144 20839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:01:02.168159 20839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:01:02.173874 20839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36195
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:01:02.175072 20839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:01:02.176126 20839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:01:02.184994 20845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:01:02.185163 20839 server_base.cc:1034] running on GCE node
W20260927 15:01:02.184993 20847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:02.185060 20844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:01:02.185843 20839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:01:02.187098 20839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:01:02.188277 20839 hybrid_clock.cc:648] HybridClock initialized: now 1790521262188253 us; error 40 us; skew 500 ppm
Sep 27 15:01:02 dist-test-slave-pz0h krb5kdc[20826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:01:02.191224 20839 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:01:02.192245 20839 webserver.cc:460] Webserver started at http://127.0.0.1:43989/ using document root <none> and password file <none>
I20260927 15:01:02.192838 20839 fs_manager.cc:359] Metadata directory not provided
I20260927 15:01:02.192910 20839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:01:02.193111 20839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:01:02.195353 20839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "4bc41f012a4e4a47b5aecfdc448fbe3b"
format_stamp: "Formatted at 2026-09-27 15:01:02 on dist-test-slave-pz0h"
server_key: "a43b17425fe9c775a2d3b3b52e18cbae"
server_key_iv: "81d93ff9d5dc438ba6e1102b24ef69d2"
server_key_version: "encryptionkey@0"
I20260927 15:01:02.195962 20839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "4bc41f012a4e4a47b5aecfdc448fbe3b"
format_stamp: "Formatted at 2026-09-27 15:01:02 on dist-test-slave-pz0h"
server_key: "a43b17425fe9c775a2d3b3b52e18cbae"
server_key_iv: "81d93ff9d5dc438ba6e1102b24ef69d2"
server_key_version: "encryptionkey@0"
I20260927 15:01:02.200095 20839 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 15:01:02.202929 20854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:01:02.204470 20839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:01:02.204627 20839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "4bc41f012a4e4a47b5aecfdc448fbe3b"
format_stamp: "Formatted at 2026-09-27 15:01:02 on dist-test-slave-pz0h"
server_key: "a43b17425fe9c775a2d3b3b52e18cbae"
server_key_iv: "81d93ff9d5dc438ba6e1102b24ef69d2"
server_key_version: "encryptionkey@0"
I20260927 15:01:02.204759 20839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:01:02.209920 20839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:01:02.213280 20839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:01:02.213470 20839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:01:02.220959 20839 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36195
I20260927 15:01:02.220973 20906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36195 every 8 connection(s)
I20260927 15:01:02.222939 20839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:01:02.226115 20907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:01:02.228950 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 20839
I20260927 15:01:02.229112 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 15:01:02.229396 392 external_mini_cluster.cc:1428] Setting key 8e113d6875c3ed5f88f9999f0432e184
I20260927 15:01:02.232806 20907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bc41f012a4e4a47b5aecfdc448fbe3b: Bootstrap starting.
I20260927 15:01:02.235039 20910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:02.230926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40865 (local address 127.0.0.1:36195)
0927 15:01:02.231481 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:02.231496 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:02.231535 (+ 39us) server_negotiation.cc:407] Connection header received
0927 15:01:02.232225 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:02.232243 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:02.232562 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:02.232900 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:02.233696 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:02.233701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:02.233714 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:01:02.233724 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:02.233778 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:01:02.233800 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:02.233925 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:02.233993 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 15:01:02.234152 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":165,"threads_started":1}
I20260927 15:01:02.236953 20907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bc41f012a4e4a47b5aecfdc448fbe3b: Neither blocks nor log segments found. Creating new log.
I20260927 15:01:02.238240 20907 log.cc:824] T 00000000000000000000000000000000 P 4bc41f012a4e4a47b5aecfdc448fbe3b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:01:02.238968 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:01:02.240609 20907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bc41f012a4e4a47b5aecfdc448fbe3b: No bootstrap required, opened a new log
I20260927 15:01:02.242674 20910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:02.241003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56779 (local address 127.0.0.1:36195)
0927 15:01:02.241130 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:02.241135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:02.241336 (+ 201us) server_negotiation.cc:407] Connection header received
0927 15:01:02.241434 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:02.241438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:02.241477 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:02.241568 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:02.242282 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:02.242286 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:02.242289 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:01:02.242302 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:02.242334 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:01:02.242347 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:02.242451 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:02.242506 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 15:01:02.242546 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:01:02.244369 20907 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bc41f012a4e4a47b5aecfdc448fbe3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc41f012a4e4a47b5aecfdc448fbe3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36195 } }
I20260927 15:01:02.244632 20907 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bc41f012a4e4a47b5aecfdc448fbe3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:01:02.244665 20907 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bc41f012a4e4a47b5aecfdc448fbe3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc41f012a4e4a47b5aecfdc448fbe3b, State: Initialized, Role: FOLLOWER
I20260927 15:01:02.245067 20907 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bc41f012a4e4a47b5aecfdc448fbe3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc41f012a4e4a47b5aecfdc448fbe3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36195 } }
I20260927 15:01:02.245187 20907 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bc41f012a4e4a47b5aecfdc448fbe3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:01:02.245229 20907 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bc41f012a4e4a47b5aecfdc448fbe3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:01:02.245303 20907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bc41f012a4e4a47b5aecfdc448fbe3b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:01:02.246552 20907 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bc41f012a4e4a47b5aecfdc448fbe3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc41f012a4e4a47b5aecfdc448fbe3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36195 } }
I20260927 15:01:02.247200 20907 leader_election.cc:302] T 00000000000000000000000000000000 P 4bc41f012a4e4a47b5aecfdc448fbe3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bc41f012a4e4a47b5aecfdc448fbe3b; no voters:
I20260927 15:01:02.247643 20907 leader_election.cc:288] T 00000000000000000000000000000000 P 4bc41f012a4e4a47b5aecfdc448fbe3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:01:02.247780 20917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bc41f012a4e4a47b5aecfdc448fbe3b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:01:02.248173 20917 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bc41f012a4e4a47b5aecfdc448fbe3b [term 1 LEADER]: Becoming Leader. State: Replica: 4bc41f012a4e4a47b5aecfdc448fbe3b, State: Running, Role: LEADER
I20260927 15:01:02.248618 20917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bc41f012a4e4a47b5aecfdc448fbe3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc41f012a4e4a47b5aecfdc448fbe3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36195 } }
I20260927 15:01:02.249286 20907 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bc41f012a4e4a47b5aecfdc448fbe3b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:01:02.250757 20918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bc41f012a4e4a47b5aecfdc448fbe3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bc41f012a4e4a47b5aecfdc448fbe3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc41f012a4e4a47b5aecfdc448fbe3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36195 } } }
I20260927 15:01:02.250973 20918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bc41f012a4e4a47b5aecfdc448fbe3b [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:02.251267 20919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bc41f012a4e4a47b5aecfdc448fbe3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bc41f012a4e4a47b5aecfdc448fbe3b. Latest consensus state: current_term: 1 leader_uuid: "4bc41f012a4e4a47b5aecfdc448fbe3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc41f012a4e4a47b5aecfdc448fbe3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36195 } } }
I20260927 15:01:02.251361 20919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bc41f012a4e4a47b5aecfdc448fbe3b [sys.catalog]: This master's current role is: LEADER
W20260927 15:01:02.254323 20932 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4bc41f012a4e4a47b5aecfdc448fbe3b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:01:02.254410 20932 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:01:02.254572 20930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:01:02.255596 20930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:01:02.261819 20930 catalog_manager.cc:1347] Generated new cluster ID: c8331a681eed468ba3c87303a9b0bbb7
I20260927 15:01:02.261910 20930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:01:02.278038 20930 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:01:02.278478 20930 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:01:02.285523 20930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bc41f012a4e4a47b5aecfdc448fbe3b: Generated new TSK 0
I20260927 15:01:02.286489 20930 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:01:02.291803 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 20839
2026-09-27T15:01:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:01:02 dist-test-slave-pz0h krb5kdc[20938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:01:02 dist-test-slave-pz0h krb5kdc[20938](info): set up 2 sockets
Sep 27 15:01:02 dist-test-slave-pz0h krb5kdc[20938](info): commencing operation
krb5kdc: starting...
W20260927 15:01:03.247081 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 15:01:03 dist-test-slave-pz0h krb5kdc[20938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:01:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:01:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:01:03.374228 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43261
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:01:03.491783 20951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:01:03.492111 20951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:01:03.492174 20951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:01:03.496481 20951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:01:03.496578 20951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:01:03.496606 20951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:01:03.496627 20951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:01:03.496654 20951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:01:03.502476 20951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:01:03.503729 20951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:01:03.504824 20951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:01:03.514106 20959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:03.514153 20956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:03.514184 20957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:01:03.514680 20951 server_base.cc:1034] running on GCE node
I20260927 15:01:03.515625 20951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:01:03.516885 20951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:01:03.518083 20951 hybrid_clock.cc:648] HybridClock initialized: now 1790521263518067 us; error 34 us; skew 500 ppm
Sep 27 15:01:03 dist-test-slave-pz0h krb5kdc[20938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:01:03.529946 20951 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:01:03.531121 20951 webserver.cc:460] Webserver started at http://127.0.0.1:43471/ using document root <none> and password file <none>
I20260927 15:01:03.531692 20951 fs_manager.cc:359] Metadata directory not provided
I20260927 15:01:03.531754 20951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:01:03.531930 20951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:01:03.534147 20951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "c069cb96e2e6431a9c2c5f06d3b09f74"
format_stamp: "Formatted at 2026-09-27 15:01:03 on dist-test-slave-pz0h"
server_key: "685622d1fc0333536a64cf18a46d6242"
server_key_iv: "b7e5baaad15d3781f2e8a4f51ed346fa"
server_key_version: "encryptionkey@0"
I20260927 15:01:03.534734 20951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "c069cb96e2e6431a9c2c5f06d3b09f74"
format_stamp: "Formatted at 2026-09-27 15:01:03 on dist-test-slave-pz0h"
server_key: "685622d1fc0333536a64cf18a46d6242"
server_key_iv: "b7e5baaad15d3781f2e8a4f51ed346fa"
server_key_version: "encryptionkey@0"
I20260927 15:01:03.539390 20951 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:01:03.542292 20966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:01:03.543735 20951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:01:03.543906 20951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "c069cb96e2e6431a9c2c5f06d3b09f74"
format_stamp: "Formatted at 2026-09-27 15:01:03 on dist-test-slave-pz0h"
server_key: "685622d1fc0333536a64cf18a46d6242"
server_key_iv: "b7e5baaad15d3781f2e8a4f51ed346fa"
server_key_version: "encryptionkey@0"
I20260927 15:01:03.544045 20951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:01:03.562781 20951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:01:03.566094 20951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:01:03.566309 20951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:01:03.573782 20951 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43261
I20260927 15:01:03.573815 21018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43261 every 8 connection(s)
I20260927 15:01:03.574847 20951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:01:03.577832 21019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:01:03.581691 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 20951
I20260927 15:01:03.581841 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 15:01:03.582106 392 external_mini_cluster.cc:1428] Setting key 427c08fbd6291979404ee5328e474868
I20260927 15:01:03.584319 21019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c069cb96e2e6431a9c2c5f06d3b09f74: Bootstrap starting.
I20260927 15:01:03.587031 21019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c069cb96e2e6431a9c2c5f06d3b09f74: Neither blocks nor log segments found. Creating new log.
I20260927 15:01:03.587838 21019 log.cc:824] T 00000000000000000000000000000000 P c069cb96e2e6431a9c2c5f06d3b09f74: Log is configured to *not* fsync() on all Append() calls
I20260927 15:01:03.590165 21019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c069cb96e2e6431a9c2c5f06d3b09f74: No bootstrap required, opened a new log
Sep 27 15:01:03 dist-test-slave-pz0h krb5kdc[20938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 15:01:03.594349 21019 raft_consensus.cc:348] T 00000000000000000000000000000000 P c069cb96e2e6431a9c2c5f06d3b09f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c069cb96e2e6431a9c2c5f06d3b09f74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43261 } }
I20260927 15:01:03.594621 21019 raft_consensus.cc:374] T 00000000000000000000000000000000 P c069cb96e2e6431a9c2c5f06d3b09f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:01:03.594657 21019 raft_consensus.cc:729] T 00000000000000000000000000000000 P c069cb96e2e6431a9c2c5f06d3b09f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c069cb96e2e6431a9c2c5f06d3b09f74, State: Initialized, Role: FOLLOWER
I20260927 15:01:03.595225 21019 consensus_queue.cc:260] T 00000000000000000000000000000000 P c069cb96e2e6431a9c2c5f06d3b09f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c069cb96e2e6431a9c2c5f06d3b09f74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43261 } }
I20260927 15:01:03.595412 21019 raft_consensus.cc:388] T 00000000000000000000000000000000 P c069cb96e2e6431a9c2c5f06d3b09f74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:01:03.595505 21019 raft_consensus.cc:482] T 00000000000000000000000000000000 P c069cb96e2e6431a9c2c5f06d3b09f74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:01:03.595647 21019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c069cb96e2e6431a9c2c5f06d3b09f74 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:01:03.597033 21019 raft_consensus.cc:504] T 00000000000000000000000000000000 P c069cb96e2e6431a9c2c5f06d3b09f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c069cb96e2e6431a9c2c5f06d3b09f74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43261 } }
I20260927 15:01:03.597470 21019 leader_election.cc:302] T 00000000000000000000000000000000 P c069cb96e2e6431a9c2c5f06d3b09f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c069cb96e2e6431a9c2c5f06d3b09f74; no voters:
I20260927 15:01:03.597836 21019 leader_election.cc:288] T 00000000000000000000000000000000 P c069cb96e2e6431a9c2c5f06d3b09f74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:01:03.597950 21024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c069cb96e2e6431a9c2c5f06d3b09f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:01:03.598191 21024 raft_consensus.cc:686] T 00000000000000000000000000000000 P c069cb96e2e6431a9c2c5f06d3b09f74 [term 1 LEADER]: Becoming Leader. State: Replica: c069cb96e2e6431a9c2c5f06d3b09f74, State: Running, Role: LEADER
I20260927 15:01:03.598412 21022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:03.583559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39713 (local address 127.0.0.1:43261)
0927 15:01:03.584009 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:03.584025 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:03.584062 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:01:03.584775 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:03.584795 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:03.585170 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:03.585771 (+ 601us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:03.586657 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:03.587856 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:03.588868 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:03.589402 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:03.589530 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:03.592537 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:03.592545 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:03.592568 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:03.592627 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:03.594602 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:03.595235 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:03.595260 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:03.595267 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:03.595451 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:03.595860 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:03.595864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:03.595866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:03.596436 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:03.596667 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:03.596986 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 15:01:03.597333 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":146,"threads_started":1}
I20260927 15:01:03.598529 21024 consensus_queue.cc:237] T 00000000000000000000000000000000 P c069cb96e2e6431a9c2c5f06d3b09f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c069cb96e2e6431a9c2c5f06d3b09f74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43261 } }
I20260927 15:01:03.599223 21019 sys_catalog.cc:564] T 00000000000000000000000000000000 P c069cb96e2e6431a9c2c5f06d3b09f74 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:01:03.600210 21025 sys_catalog.cc:455] T 00000000000000000000000000000000 P c069cb96e2e6431a9c2c5f06d3b09f74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c069cb96e2e6431a9c2c5f06d3b09f74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c069cb96e2e6431a9c2c5f06d3b09f74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43261 } } }
I20260927 15:01:03.600349 21025 sys_catalog.cc:458] T 00000000000000000000000000000000 P c069cb96e2e6431a9c2c5f06d3b09f74 [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:03.600629 21026 sys_catalog.cc:455] T 00000000000000000000000000000000 P c069cb96e2e6431a9c2c5f06d3b09f74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c069cb96e2e6431a9c2c5f06d3b09f74. Latest consensus state: current_term: 1 leader_uuid: "c069cb96e2e6431a9c2c5f06d3b09f74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c069cb96e2e6431a9c2c5f06d3b09f74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43261 } } }
I20260927 15:01:03.600742 21026 sys_catalog.cc:458] T 00000000000000000000000000000000 P c069cb96e2e6431a9c2c5f06d3b09f74 [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:03.601765 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 15:01:03.604300 21039 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c069cb96e2e6431a9c2c5f06d3b09f74: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:01:03.604393 21039 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:01:03.604457 21033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:01:03.605180 21033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:01:03.611406 21033 catalog_manager.cc:1347] Generated new cluster ID: 3d106016a65d41b29bff341652457b6a
I20260927 15:01:03.611488 21033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:01:03.611785 21022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:03.603934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35987 (local address 127.0.0.1:43261)
0927 15:01:03.604045 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:03.604049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:03.604248 (+ 199us) server_negotiation.cc:407] Connection header received
0927 15:01:03.604360 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:03.604363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:03.604411 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:03.604476 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:03.605197 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:03.605786 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:03.606574 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:03.606883 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:03.606979 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:03.608301 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:03.608305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:03.608308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:03.608335 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:03.609917 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:03.610325 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:03.610328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:03.610330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:03.610412 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:03.610652 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:03.610655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:03.610657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:03.610926 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:03.611054 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:03.611505 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 15:01:03.611641 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:01:03.625993 21033 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:01:03.627467 21033 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:01:03.635996 21033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c069cb96e2e6431a9c2c5f06d3b09f74: Generated new TSK 0
I20260927 15:01:03.637012 21033 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:01:03.642968 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 20951
2026-09-27T15:01:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1351 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:01:03 dist-test-slave-pz0h krb5kdc[21050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:01:03 dist-test-slave-pz0h krb5kdc[21050](info): set up 2 sockets
Sep 27 15:01:03 dist-test-slave-pz0h krb5kdc[21050](info): commencing operation
krb5kdc: starting...
W20260927 15:01:04.606825 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 15:01:04 dist-test-slave-pz0h krb5kdc[21050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:01:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:01:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:01:04.726223 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37203
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:01:04.844576 21063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:01:04.844909 21063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:01:04.844982 21063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:01:04.849267 21063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:01:04.849504 21063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:01:04.849573 21063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:01:04.849607 21063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:01:04.849634 21063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:01:04.855654 21063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:01:04.856827 21063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:01:04.857846 21063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:01:04.866947 21071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:01:04.867039 21063 server_base.cc:1034] running on GCE node
W20260927 15:01:04.866945 21069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:04.866961 21068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:01:04.867848 21063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:01:04.869236 21063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:01:04.870510 21063 hybrid_clock.cc:648] HybridClock initialized: now 1790521264870473 us; error 58 us; skew 500 ppm
Sep 27 15:01:04 dist-test-slave-pz0h krb5kdc[21050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:01:04.873847 21063 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:01:04.874953 21063 webserver.cc:460] Webserver started at http://127.0.0.1:43699/ using document root <none> and password file <none>
I20260927 15:01:04.875554 21063 fs_manager.cc:359] Metadata directory not provided
I20260927 15:01:04.875631 21063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:01:04.875870 21063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:01:04.878116 21063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "54f8ee1539a345fca6fa7d41e65d6ae9"
format_stamp: "Formatted at 2026-09-27 15:01:04 on dist-test-slave-pz0h"
server_key: "c51f70b6084d1a53c2fdde6fdd741d34"
server_key_iv: "63edc1c2983809035cd20ae50b4df1d4"
server_key_version: "encryptionkey@0"
I20260927 15:01:04.878742 21063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "54f8ee1539a345fca6fa7d41e65d6ae9"
format_stamp: "Formatted at 2026-09-27 15:01:04 on dist-test-slave-pz0h"
server_key: "c51f70b6084d1a53c2fdde6fdd741d34"
server_key_iv: "63edc1c2983809035cd20ae50b4df1d4"
server_key_version: "encryptionkey@0"
I20260927 15:01:04.883363 21063 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260927 15:01:04.886214 21078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:01:04.887946 21063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:01:04.888115 21063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "54f8ee1539a345fca6fa7d41e65d6ae9"
format_stamp: "Formatted at 2026-09-27 15:01:04 on dist-test-slave-pz0h"
server_key: "c51f70b6084d1a53c2fdde6fdd741d34"
server_key_iv: "63edc1c2983809035cd20ae50b4df1d4"
server_key_version: "encryptionkey@0"
I20260927 15:01:04.888254 21063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:01:04.905094 21063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:01:04.908420 21063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:01:04.908624 21063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:01:04.916218 21063 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37203
I20260927 15:01:04.916239 21130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37203 every 8 connection(s)
I20260927 15:01:04.917248 21063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:01:04.920228 21131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:01:04.923465 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 21063
I20260927 15:01:04.923667 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 15:01:04.924055 392 external_mini_cluster.cc:1428] Setting key ef355a9c22673079e8d7f445f75e371e
I20260927 15:01:04.926919 21131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54f8ee1539a345fca6fa7d41e65d6ae9: Bootstrap starting.
I20260927 15:01:04.929833 21131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54f8ee1539a345fca6fa7d41e65d6ae9: Neither blocks nor log segments found. Creating new log.
I20260927 15:01:04.930668 21131 log.cc:824] T 00000000000000000000000000000000 P 54f8ee1539a345fca6fa7d41e65d6ae9: Log is configured to *not* fsync() on all Append() calls
I20260927 15:01:04.933228 21131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54f8ee1539a345fca6fa7d41e65d6ae9: No bootstrap required, opened a new log
I20260927 15:01:04.934856 21134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:04.925630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47839 (local address 127.0.0.1:37203)
0927 15:01:04.926100 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:04.926121 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:04.926177 (+ 56us) server_negotiation.cc:407] Connection header received
0927 15:01:04.927074 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:04.927098 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:04.927561 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:04.927933 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:04.928409 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:04.929845 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:04.930873 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:04.931576 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:04.931753 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:04.932654 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:04.932660 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:04.932678 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:01:04.933175 (+ 497us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:04.933249 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:01:04.933283 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:04.933384 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:04.933569 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 15:01:04.933840 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":126,"threads_started":1}
I20260927 15:01:04.937743 21131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54f8ee1539a345fca6fa7d41e65d6ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f8ee1539a345fca6fa7d41e65d6ae9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37203 } }
I20260927 15:01:04.938068 21131 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54f8ee1539a345fca6fa7d41e65d6ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:01:04.938112 21131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54f8ee1539a345fca6fa7d41e65d6ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f8ee1539a345fca6fa7d41e65d6ae9, State: Initialized, Role: FOLLOWER
I20260927 15:01:04.938087 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:01:04.938625 21131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54f8ee1539a345fca6fa7d41e65d6ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f8ee1539a345fca6fa7d41e65d6ae9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37203 } }
I20260927 15:01:04.938822 21131 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54f8ee1539a345fca6fa7d41e65d6ae9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:01:04.938898 21131 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54f8ee1539a345fca6fa7d41e65d6ae9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:01:04.939004 21131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54f8ee1539a345fca6fa7d41e65d6ae9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:01:04.940299 21131 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54f8ee1539a345fca6fa7d41e65d6ae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f8ee1539a345fca6fa7d41e65d6ae9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37203 } }
I20260927 15:01:04.940753 21131 leader_election.cc:302] T 00000000000000000000000000000000 P 54f8ee1539a345fca6fa7d41e65d6ae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54f8ee1539a345fca6fa7d41e65d6ae9; no voters:
I20260927 15:01:04.941136 21131 leader_election.cc:288] T 00000000000000000000000000000000 P 54f8ee1539a345fca6fa7d41e65d6ae9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:01:04.941233 21141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54f8ee1539a345fca6fa7d41e65d6ae9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:01:04.941437 21141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54f8ee1539a345fca6fa7d41e65d6ae9 [term 1 LEADER]: Becoming Leader. State: Replica: 54f8ee1539a345fca6fa7d41e65d6ae9, State: Running, Role: LEADER
I20260927 15:01:04.941803 21141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54f8ee1539a345fca6fa7d41e65d6ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f8ee1539a345fca6fa7d41e65d6ae9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37203 } }
I20260927 15:01:04.942492 21131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54f8ee1539a345fca6fa7d41e65d6ae9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:01:04.943590 21142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54f8ee1539a345fca6fa7d41e65d6ae9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54f8ee1539a345fca6fa7d41e65d6ae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f8ee1539a345fca6fa7d41e65d6ae9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37203 } } }
I20260927 15:01:04.943729 21142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54f8ee1539a345fca6fa7d41e65d6ae9 [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:04.944453 21143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54f8ee1539a345fca6fa7d41e65d6ae9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54f8ee1539a345fca6fa7d41e65d6ae9. Latest consensus state: current_term: 1 leader_uuid: "54f8ee1539a345fca6fa7d41e65d6ae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f8ee1539a345fca6fa7d41e65d6ae9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37203 } } }
I20260927 15:01:04.945350 21134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:04.940294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37803 (local address 127.0.0.1:37203)
0927 15:01:04.940425 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:04.940431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:04.940520 (+ 89us) server_negotiation.cc:407] Connection header received
0927 15:01:04.940637 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:04.940640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:04.940690 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:04.940784 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:04.941076 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:04.941828 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:04.943008 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:04.943342 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:04.943440 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:04.944519 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:04.944523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:04.944525 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:01:04.944539 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:04.944569 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:01:04.944582 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:04.944645 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:04.945136 (+ 491us) server_negotiation.cc:299] Negotiation successful
0927 15:01:04.945198 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:01:04.945271 21143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54f8ee1539a345fca6fa7d41e65d6ae9 [sys.catalog]: This master's current role is: LEADER
W20260927 15:01:04.947683 21156 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 54f8ee1539a345fca6fa7d41e65d6ae9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:01:04.947758 21156 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:01:04.947836 21150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:01:04.948568 21150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:01:04.955302 21150 catalog_manager.cc:1347] Generated new cluster ID: 1f87125bd5834791abb0c61e953a57d4
I20260927 15:01:04.955399 21150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:01:04.968228 21150 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:01:04.969307 21150 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:01:04.978375 21150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54f8ee1539a345fca6fa7d41e65d6ae9: Generated new TSK 0
I20260927 15:01:04.979152 21150 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:01:04.990597 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 21063
2026-09-27T15:01:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:01:05 dist-test-slave-pz0h krb5kdc[21162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:01:05 dist-test-slave-pz0h krb5kdc[21162](info): set up 2 sockets
Sep 27 15:01:05 dist-test-slave-pz0h krb5kdc[21162](info): commencing operation
krb5kdc: starting...
W20260927 15:01:05.947279 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 15:01:06 dist-test-slave-pz0h krb5kdc[21162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:01:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:01:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:01:06.065212 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33997
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:01:06.177992 21175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:01:06.178287 21175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:01:06.178330 21175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:01:06.182561 21175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:01:06.182641 21175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:01:06.182659 21175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:01:06.182674 21175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:01:06.182688 21175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:01:06.188633 21175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:01:06.189738 21175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:01:06.190774 21175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:01:06.199360 21180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:01:06.199457 21175 server_base.cc:1034] running on GCE node
W20260927 15:01:06.199366 21183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:06.199357 21181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:01:06.200157 21175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:01:06.201336 21175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:01:06.202494 21175 hybrid_clock.cc:648] HybridClock initialized: now 1790521266202479 us; error 33 us; skew 500 ppm
Sep 27 15:01:06 dist-test-slave-pz0h krb5kdc[21162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:01:06.205250 21175 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:01:06.206224 21175 webserver.cc:460] Webserver started at http://127.0.0.1:40201/ using document root <none> and password file <none>
I20260927 15:01:06.206841 21175 fs_manager.cc:359] Metadata directory not provided
I20260927 15:01:06.206931 21175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:01:06.207149 21175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:01:06.209378 21175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "51facd6eff5b4ef28c6d1323a6575f5d"
format_stamp: "Formatted at 2026-09-27 15:01:06 on dist-test-slave-pz0h"
server_key: "5b75f456691f905f4f09be1166be694e"
server_key_iv: "b9357b5500e564ef0e254909c0579916"
server_key_version: "encryptionkey@0"
I20260927 15:01:06.210003 21175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "51facd6eff5b4ef28c6d1323a6575f5d"
format_stamp: "Formatted at 2026-09-27 15:01:06 on dist-test-slave-pz0h"
server_key: "5b75f456691f905f4f09be1166be694e"
server_key_iv: "b9357b5500e564ef0e254909c0579916"
server_key_version: "encryptionkey@0"
I20260927 15:01:06.214351 21175 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 15:01:06.217232 21190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:01:06.218591 21175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:01:06.218729 21175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "51facd6eff5b4ef28c6d1323a6575f5d"
format_stamp: "Formatted at 2026-09-27 15:01:06 on dist-test-slave-pz0h"
server_key: "5b75f456691f905f4f09be1166be694e"
server_key_iv: "b9357b5500e564ef0e254909c0579916"
server_key_version: "encryptionkey@0"
I20260927 15:01:06.218916 21175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:01:06.223883 21175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:01:06.227190 21175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:01:06.227474 21175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:01:06.234706 21175 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33997
I20260927 15:01:06.234714 21242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33997 every 8 connection(s)
I20260927 15:01:06.235743 21175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:01:06.238600 21243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:01:06.241657 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 21175
I20260927 15:01:06.241803 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 15:01:06.242040 392 external_mini_cluster.cc:1428] Setting key 715fde7c4335ba756523943b4c944364
I20260927 15:01:06.244696 21243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51facd6eff5b4ef28c6d1323a6575f5d: Bootstrap starting.
I20260927 15:01:06.247189 21243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51facd6eff5b4ef28c6d1323a6575f5d: Neither blocks nor log segments found. Creating new log.
I20260927 15:01:06.247202 21246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:06.243431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48419 (local address 127.0.0.1:33997)
0927 15:01:06.243873 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:06.243890 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:06.243927 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:01:06.244555 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:06.244572 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:06.244894 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:06.245181 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:06.245983 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:06.245988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:06.246000 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:01:06.246010 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:06.246069 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:01:06.246088 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:06.246193 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:06.246281 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 15:01:06.246433 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":134,"threads_started":1}
I20260927 15:01:06.247936 21243 log.cc:824] T 00000000000000000000000000000000 P 51facd6eff5b4ef28c6d1323a6575f5d: Log is configured to *not* fsync() on all Append() calls
I20260927 15:01:06.249912 21243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51facd6eff5b4ef28c6d1323a6575f5d: No bootstrap required, opened a new log
I20260927 15:01:06.250098 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:01:06.253567 21246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:06.252064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33815 (local address 127.0.0.1:33997)
0927 15:01:06.252172 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:06.252176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:06.252332 (+ 156us) server_negotiation.cc:407] Connection header received
0927 15:01:06.252441 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:06.252444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:06.252479 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:06.252543 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:06.253205 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:06.253208 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:06.253211 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:01:06.253223 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:06.253251 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:01:06.253262 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:06.253333 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:06.253409 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 15:01:06.253444 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 15:01:06.253783 21243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51facd6eff5b4ef28c6d1323a6575f5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51facd6eff5b4ef28c6d1323a6575f5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33997 } }
I20260927 15:01:06.254072 21243 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51facd6eff5b4ef28c6d1323a6575f5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:01:06.254132 21243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51facd6eff5b4ef28c6d1323a6575f5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51facd6eff5b4ef28c6d1323a6575f5d, State: Initialized, Role: FOLLOWER
I20260927 15:01:06.254585 21243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51facd6eff5b4ef28c6d1323a6575f5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51facd6eff5b4ef28c6d1323a6575f5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33997 } }
I20260927 15:01:06.254724 21243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51facd6eff5b4ef28c6d1323a6575f5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:01:06.254808 21243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51facd6eff5b4ef28c6d1323a6575f5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:01:06.254910 21243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51facd6eff5b4ef28c6d1323a6575f5d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:01:06.255874 21243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51facd6eff5b4ef28c6d1323a6575f5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51facd6eff5b4ef28c6d1323a6575f5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33997 } }
I20260927 15:01:06.256218 21243 leader_election.cc:302] T 00000000000000000000000000000000 P 51facd6eff5b4ef28c6d1323a6575f5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51facd6eff5b4ef28c6d1323a6575f5d; no voters:
I20260927 15:01:06.256534 21243 leader_election.cc:288] T 00000000000000000000000000000000 P 51facd6eff5b4ef28c6d1323a6575f5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:01:06.256623 21253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51facd6eff5b4ef28c6d1323a6575f5d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:01:06.256821 21253 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51facd6eff5b4ef28c6d1323a6575f5d [term 1 LEADER]: Becoming Leader. State: Replica: 51facd6eff5b4ef28c6d1323a6575f5d, State: Running, Role: LEADER
I20260927 15:01:06.257166 21253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51facd6eff5b4ef28c6d1323a6575f5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51facd6eff5b4ef28c6d1323a6575f5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33997 } }
I20260927 15:01:06.257737 21243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51facd6eff5b4ef28c6d1323a6575f5d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:01:06.258698 21255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51facd6eff5b4ef28c6d1323a6575f5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51facd6eff5b4ef28c6d1323a6575f5d. Latest consensus state: current_term: 1 leader_uuid: "51facd6eff5b4ef28c6d1323a6575f5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51facd6eff5b4ef28c6d1323a6575f5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33997 } } }
I20260927 15:01:06.258858 21255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51facd6eff5b4ef28c6d1323a6575f5d [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:06.259265 21262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:01:06.259276 21254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51facd6eff5b4ef28c6d1323a6575f5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51facd6eff5b4ef28c6d1323a6575f5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51facd6eff5b4ef28c6d1323a6575f5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33997 } } }
I20260927 15:01:06.259459 21254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51facd6eff5b4ef28c6d1323a6575f5d [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:06.262145 21262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:01:06.267546 21262 catalog_manager.cc:1347] Generated new cluster ID: 53144328b6f442819d43c248509f207c
I20260927 15:01:06.267630 21262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:01:06.277201 21262 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:01:06.277532 21262 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:01:06.288338 21262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51facd6eff5b4ef28c6d1323a6575f5d: Generated new TSK 0
I20260927 15:01:06.289139 21262 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:01:06.301689 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 21175
2026-09-27T15:01:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:01:06 dist-test-slave-pz0h krb5kdc[21274](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:01:06 dist-test-slave-pz0h krb5kdc[21274](info): set up 2 sockets
Sep 27 15:01:06 dist-test-slave-pz0h krb5kdc[21274](info): commencing operation
krb5kdc: starting...
W20260927 15:01:07.250782 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 15:01:07 dist-test-slave-pz0h krb5kdc[21274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:01:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:01:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:01:07.375147 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45647
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:01:07.493884 21287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:01:07.494186 21287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:01:07.494230 21287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:01:07.498458 21287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:01:07.498551 21287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:01:07.498569 21287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:01:07.498586 21287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:01:07.498600 21287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:01:07.504338 21287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:01:07.505511 21287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:01:07.506537 21287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:01:07.515159 21293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:07.515270 21295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:07.515159 21292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:01:07.515446 21287 server_base.cc:1034] running on GCE node
I20260927 15:01:07.516527 21287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:01:07.517859 21287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:01:07.519084 21287 hybrid_clock.cc:648] HybridClock initialized: now 1790521267519055 us; error 52 us; skew 500 ppm
Sep 27 15:01:07 dist-test-slave-pz0h krb5kdc[21274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:01:07.522185 21287 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:01:07.523272 21287 webserver.cc:460] Webserver started at http://127.0.0.1:43361/ using document root <none> and password file <none>
I20260927 15:01:07.523854 21287 fs_manager.cc:359] Metadata directory not provided
I20260927 15:01:07.523917 21287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:01:07.524101 21287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:01:07.526222 21287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "4165b856cf894da781b203a8dbc73a27"
format_stamp: "Formatted at 2026-09-27 15:01:07 on dist-test-slave-pz0h"
server_key: "4f82080d2d65086c02ba05e3da1729f3"
server_key_iv: "08c42e205fd10de7f74bb1a18d6632cc"
server_key_version: "encryptionkey@0"
I20260927 15:01:07.526787 21287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "4165b856cf894da781b203a8dbc73a27"
format_stamp: "Formatted at 2026-09-27 15:01:07 on dist-test-slave-pz0h"
server_key: "4f82080d2d65086c02ba05e3da1729f3"
server_key_iv: "08c42e205fd10de7f74bb1a18d6632cc"
server_key_version: "encryptionkey@0"
I20260927 15:01:07.530700 21287 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 15:01:07.533495 21302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:01:07.534736 21287 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:01:07.534907 21287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "4165b856cf894da781b203a8dbc73a27"
format_stamp: "Formatted at 2026-09-27 15:01:07 on dist-test-slave-pz0h"
server_key: "4f82080d2d65086c02ba05e3da1729f3"
server_key_iv: "08c42e205fd10de7f74bb1a18d6632cc"
server_key_version: "encryptionkey@0"
I20260927 15:01:07.535041 21287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:01:07.549525 21287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:01:07.552899 21287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:01:07.553092 21287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:01:07.561498 21287 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45647
I20260927 15:01:07.561550 21354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45647 every 8 connection(s)
I20260927 15:01:07.562597 21287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:01:07.563007 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 21287
I20260927 15:01:07.563138 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 15:01:07.563387 392 external_mini_cluster.cc:1428] Setting key 65a82227074f224628902fc9f03d03d9
I20260927 15:01:07.565798 21355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 15:01:07 dist-test-slave-pz0h krb5kdc[21274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 15:01:07.572676 21355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4165b856cf894da781b203a8dbc73a27: Bootstrap starting.
I20260927 15:01:07.575572 21355 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4165b856cf894da781b203a8dbc73a27: Neither blocks nor log segments found. Creating new log.
I20260927 15:01:07.576429 21355 log.cc:824] T 00000000000000000000000000000000 P 4165b856cf894da781b203a8dbc73a27: Log is configured to *not* fsync() on all Append() calls
I20260927 15:01:07.578493 21358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:07.564708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.153:54309 (local address 127.0.0.1:45647)
0927 15:01:07.565160 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:07.565182 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:07.565226 (+ 44us) server_negotiation.cc:407] Connection header received
0927 15:01:07.565881 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:07.565900 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:07.566276 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:07.566576 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:07.567492 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:07.568516 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:07.569668 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:07.570159 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:07.570283 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:07.573128 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:07.573135 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:07.573154 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:07.573193 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:07.575194 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:07.575833 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:07.575839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:07.575844 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:07.575982 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:07.576350 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:07.576353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:07.576354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:07.576827 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:07.577078 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:07.577336 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 15:01:07.577591 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":135,"threads_started":1}
I20260927 15:01:07.578655 21355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4165b856cf894da781b203a8dbc73a27: No bootstrap required, opened a new log
I20260927 15:01:07.581804 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:01:07.583168 21355 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4165b856cf894da781b203a8dbc73a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4165b856cf894da781b203a8dbc73a27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45647 } }
I20260927 15:01:07.583529 21355 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4165b856cf894da781b203a8dbc73a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:01:07.583600 21355 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4165b856cf894da781b203a8dbc73a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4165b856cf894da781b203a8dbc73a27, State: Initialized, Role: FOLLOWER
I20260927 15:01:07.584072 21355 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4165b856cf894da781b203a8dbc73a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4165b856cf894da781b203a8dbc73a27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45647 } }
I20260927 15:01:07.584275 21355 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4165b856cf894da781b203a8dbc73a27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:01:07.584357 21355 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4165b856cf894da781b203a8dbc73a27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:01:07.584465 21355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4165b856cf894da781b203a8dbc73a27 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:01:07.585600 21355 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4165b856cf894da781b203a8dbc73a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4165b856cf894da781b203a8dbc73a27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45647 } }
I20260927 15:01:07.585978 21355 leader_election.cc:302] T 00000000000000000000000000000000 P 4165b856cf894da781b203a8dbc73a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4165b856cf894da781b203a8dbc73a27; no voters:
I20260927 15:01:07.586387 21355 leader_election.cc:288] T 00000000000000000000000000000000 P 4165b856cf894da781b203a8dbc73a27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:01:07.586498 21365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4165b856cf894da781b203a8dbc73a27 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:01:07.586702 21365 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4165b856cf894da781b203a8dbc73a27 [term 1 LEADER]: Becoming Leader. State: Replica: 4165b856cf894da781b203a8dbc73a27, State: Running, Role: LEADER
I20260927 15:01:07.587096 21365 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4165b856cf894da781b203a8dbc73a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4165b856cf894da781b203a8dbc73a27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45647 } }
I20260927 15:01:07.587842 21355 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4165b856cf894da781b203a8dbc73a27 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:01:07.588662 21367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4165b856cf894da781b203a8dbc73a27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4165b856cf894da781b203a8dbc73a27. Latest consensus state: current_term: 1 leader_uuid: "4165b856cf894da781b203a8dbc73a27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4165b856cf894da781b203a8dbc73a27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45647 } } }
I20260927 15:01:07.588802 21367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4165b856cf894da781b203a8dbc73a27 [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:07.589404 21366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4165b856cf894da781b203a8dbc73a27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4165b856cf894da781b203a8dbc73a27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4165b856cf894da781b203a8dbc73a27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45647 } } }
I20260927 15:01:07.589504 21366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4165b856cf894da781b203a8dbc73a27 [sys.catalog]: This master's current role is: LEADER
W20260927 15:01:07.593214 21382 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4165b856cf894da781b203a8dbc73a27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:01:07.593304 21382 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:01:07.593389 21374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:01:07.594070 21358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:07.584132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.153:39777 (local address 127.0.0.1:45647)
0927 15:01:07.584265 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:07.584270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:07.584511 (+ 241us) server_negotiation.cc:407] Connection header received
0927 15:01:07.584655 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:07.584658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:07.584712 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:07.584805 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:07.586000 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:07.586738 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:07.588012 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:07.588334 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:07.588435 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:07.590346 (+ 1911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:07.590349 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:07.590351 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:07.590376 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:07.591851 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:07.592427 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:07.592431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:07.592433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:07.592525 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:07.592863 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:07.592866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:07.592868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:07.593097 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:07.593241 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:07.593824 (+ 583us) server_negotiation.cc:299] Negotiation successful
0927 15:01:07.593926 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:01:07.594551 21374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:01:07.602840 21374 catalog_manager.cc:1347] Generated new cluster ID: a9b628e4181b48979b2658190c88f365
I20260927 15:01:07.602931 21374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:01:07.619124 21374 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:01:07.621224 21374 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:01:07.631978 21374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4165b856cf894da781b203a8dbc73a27: Generated new TSK 0
I20260927 15:01:07.633039 21374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:01:07.643062 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 21287
W20260927 15:01:07.647992 21272 connection.cc:535] client connection to 127.0.0.1:45647 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45647): Connection reset by peer (error 104)
2026-09-27T15:01:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:01:07 dist-test-slave-pz0h krb5kdc[21387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:01:07 dist-test-slave-pz0h krb5kdc[21387](info): set up 2 sockets
Sep 27 15:01:07 dist-test-slave-pz0h krb5kdc[21387](info): commencing operation
krb5kdc: starting...
W20260927 15:01:08.602898 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 15:01:08 dist-test-slave-pz0h krb5kdc[21387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:01:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:01:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:01:08.729892 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38231
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:01:08.850839 21400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:01:08.851155 21400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:01:08.851217 21400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:01:08.855527 21400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:01:08.855625 21400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:01:08.855652 21400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:01:08.855676 21400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:01:08.855698 21400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:01:08.861487 21400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:01:08.862627 21400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:01:08.863699 21400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:01:08.872565 21400 server_base.cc:1034] running on GCE node
W20260927 15:01:08.872522 21406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:08.872522 21408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:08.872527 21405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:01:08.873394 21400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:01:08.874606 21400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:01:08.875808 21400 hybrid_clock.cc:648] HybridClock initialized: now 1790521268875782 us; error 46 us; skew 500 ppm
Sep 27 15:01:08 dist-test-slave-pz0h krb5kdc[21387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:01:08.883035 21400 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:01:08.884261 21400 webserver.cc:460] Webserver started at http://127.0.0.1:40357/ using document root <none> and password file <none>
I20260927 15:01:08.884840 21400 fs_manager.cc:359] Metadata directory not provided
I20260927 15:01:08.884907 21400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:01:08.885098 21400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:01:08.887497 21400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "c4c1c9198d914bc7b941107597642a35"
format_stamp: "Formatted at 2026-09-27 15:01:08 on dist-test-slave-pz0h"
server_key: "a049e3d503015942a0904249f16b0b71"
server_key_iv: "69909cc9b62ffcc302d7ab4416cdf38a"
server_key_version: "encryptionkey@0"
I20260927 15:01:08.888082 21400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "c4c1c9198d914bc7b941107597642a35"
format_stamp: "Formatted at 2026-09-27 15:01:08 on dist-test-slave-pz0h"
server_key: "a049e3d503015942a0904249f16b0b71"
server_key_iv: "69909cc9b62ffcc302d7ab4416cdf38a"
server_key_version: "encryptionkey@0"
I20260927 15:01:08.892096 21400 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 15:01:08.894697 21415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:01:08.896121 21400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:01:08.896274 21400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "c4c1c9198d914bc7b941107597642a35"
format_stamp: "Formatted at 2026-09-27 15:01:08 on dist-test-slave-pz0h"
server_key: "a049e3d503015942a0904249f16b0b71"
server_key_iv: "69909cc9b62ffcc302d7ab4416cdf38a"
server_key_version: "encryptionkey@0"
I20260927 15:01:08.896431 21400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:01:08.939728 21400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:01:08.943890 21400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:01:08.944070 21400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:01:08.951277 21400 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38231
I20260927 15:01:08.951334 21467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38231 every 8 connection(s)
I20260927 15:01:08.952303 21400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:01:08.955229 21468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:01:08.957479 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 21400
I20260927 15:01:08.957602 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 15:01:08.957867 392 external_mini_cluster.cc:1428] Setting key 8a63c9ff292b73688aba6863db41215b
I20260927 15:01:08.961995 21468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4c1c9198d914bc7b941107597642a35: Bootstrap starting.
I20260927 15:01:08.964926 21468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4c1c9198d914bc7b941107597642a35: Neither blocks nor log segments found. Creating new log.
I20260927 15:01:08.965883 21468 log.cc:824] T 00000000000000000000000000000000 P c4c1c9198d914bc7b941107597642a35: Log is configured to *not* fsync() on all Append() calls
I20260927 15:01:08.968500 21468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4c1c9198d914bc7b941107597642a35: No bootstrap required, opened a new log
I20260927 15:01:08.968559 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:08.959302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.153:33457 (local address 127.0.0.1:38231)
0927 15:01:08.959774 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:08.959794 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:08.959840 (+ 46us) server_negotiation.cc:407] Connection header received
0927 15:01:08.960709 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:08.960736 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:08.961217 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:08.961614 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:08.962105 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:08.963639 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:08.964628 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:08.965351 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:08.965517 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:08.966395 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:08.966400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:08.966415 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:01:08.966907 (+ 492us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:08.966979 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:01:08.967011 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:08.967146 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:08.967347 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 15:01:08.967495 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":140,"threads_started":1}
I20260927 15:01:08.971858 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:01:08.972632 21468 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4c1c9198d914bc7b941107597642a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c1c9198d914bc7b941107597642a35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38231 } }
I20260927 15:01:08.972993 21468 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4c1c9198d914bc7b941107597642a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:01:08.973049 21468 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4c1c9198d914bc7b941107597642a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4c1c9198d914bc7b941107597642a35, State: Initialized, Role: FOLLOWER
I20260927 15:01:08.973608 21468 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4c1c9198d914bc7b941107597642a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c1c9198d914bc7b941107597642a35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38231 } }
I20260927 15:01:08.973814 21468 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4c1c9198d914bc7b941107597642a35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:01:08.973886 21468 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4c1c9198d914bc7b941107597642a35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:01:08.973976 21468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4c1c9198d914bc7b941107597642a35 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:01:08.975322 21468 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4c1c9198d914bc7b941107597642a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c1c9198d914bc7b941107597642a35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38231 } }
I20260927 15:01:08.975764 21468 leader_election.cc:302] T 00000000000000000000000000000000 P c4c1c9198d914bc7b941107597642a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4c1c9198d914bc7b941107597642a35; no voters:
I20260927 15:01:08.976116 21468 leader_election.cc:288] T 00000000000000000000000000000000 P c4c1c9198d914bc7b941107597642a35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:01:08.976207 21478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4c1c9198d914bc7b941107597642a35 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:01:08.976409 21478 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4c1c9198d914bc7b941107597642a35 [term 1 LEADER]: Becoming Leader. State: Replica: c4c1c9198d914bc7b941107597642a35, State: Running, Role: LEADER
I20260927 15:01:08.976826 21478 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4c1c9198d914bc7b941107597642a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c1c9198d914bc7b941107597642a35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38231 } }
I20260927 15:01:08.977660 21468 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4c1c9198d914bc7b941107597642a35 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:01:08.978576 21479 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4c1c9198d914bc7b941107597642a35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4c1c9198d914bc7b941107597642a35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c1c9198d914bc7b941107597642a35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38231 } } }
I20260927 15:01:08.978737 21479 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4c1c9198d914bc7b941107597642a35 [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:08.979104 21480 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4c1c9198d914bc7b941107597642a35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4c1c9198d914bc7b941107597642a35. Latest consensus state: current_term: 1 leader_uuid: "c4c1c9198d914bc7b941107597642a35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c1c9198d914bc7b941107597642a35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38231 } } }
I20260927 15:01:08.979187 21480 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4c1c9198d914bc7b941107597642a35 [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:08.979947 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:08.974629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.153:50881 (local address 127.0.0.1:38231)
0927 15:01:08.974862 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:08.974868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:08.975125 (+ 257us) server_negotiation.cc:407] Connection header received
0927 15:01:08.975301 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:08.975305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:08.975361 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:08.975461 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:08.975795 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:08.976450 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:08.977444 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:08.977698 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:08.978194 (+ 496us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:08.979445 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:08.979449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:08.979452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:01:08.979467 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:08.979499 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:01:08.979515 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:08.979595 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:08.979774 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 15:01:08.979813 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 15:01:08.980154 21487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:01:08.983331 21487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:01:08.989737 21487 catalog_manager.cc:1347] Generated new cluster ID: 4f64a30021ad4995886bb0e7d13959f1
I20260927 15:01:08.989833 21487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:01:09.000187 21487 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:01:09.001555 21487 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:01:09.016993 21487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4c1c9198d914bc7b941107597642a35: Generated new TSK 0
I20260927 15:01:09.017823 21487 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:01:09.029742 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 21400
W20260927 15:01:09.034533 21385 connection.cc:535] client connection to 127.0.0.1:38231 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38231): Connection reset by peer (error 104)
2026-09-27T15:01:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:01:09 dist-test-slave-pz0h krb5kdc[21499](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:01:09 dist-test-slave-pz0h krb5kdc[21499](info): set up 2 sockets
Sep 27 15:01:09 dist-test-slave-pz0h krb5kdc[21499](info): commencing operation
krb5kdc: starting...
W20260927 15:01:09.982880 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 15:01:10 dist-test-slave-pz0h krb5kdc[21499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:01:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:01:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:01:10.108319 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40237
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:01:10.224522 21512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:01:10.224825 21512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:01:10.224871 21512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:01:10.229107 21512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:01:10.229200 21512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:01:10.229219 21512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:01:10.229236 21512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:01:10.229250 21512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:01:10.234947 21512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:01:10.236045 21512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:01:10.237098 21512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:01:10.245751 21520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:10.245759 21517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:10.245759 21518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:01:10.246145 21512 server_base.cc:1034] running on GCE node
I20260927 15:01:10.246966 21512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:01:10.248270 21512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:01:10.249485 21512 hybrid_clock.cc:648] HybridClock initialized: now 1790521270249469 us; error 36 us; skew 500 ppm
Sep 27 15:01:10 dist-test-slave-pz0h krb5kdc[21499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:01:10.252468 21512 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:01:10.253604 21512 webserver.cc:460] Webserver started at http://127.0.0.1:41873/ using document root <none> and password file <none>
I20260927 15:01:10.254225 21512 fs_manager.cc:359] Metadata directory not provided
I20260927 15:01:10.254304 21512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:01:10.254525 21512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:01:10.256767 21512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "b70982305d8542b2aa0b02a1432868bb"
format_stamp: "Formatted at 2026-09-27 15:01:10 on dist-test-slave-pz0h"
server_key: "3517e6320b8a9a79102c22e4fec215cd"
server_key_iv: "9f8d24ad6346e91c03518e8a5c356876"
server_key_version: "encryptionkey@0"
I20260927 15:01:10.257359 21512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "b70982305d8542b2aa0b02a1432868bb"
format_stamp: "Formatted at 2026-09-27 15:01:10 on dist-test-slave-pz0h"
server_key: "3517e6320b8a9a79102c22e4fec215cd"
server_key_iv: "9f8d24ad6346e91c03518e8a5c356876"
server_key_version: "encryptionkey@0"
I20260927 15:01:10.261507 21512 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:01:10.264252 21527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:01:10.265678 21512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:01:10.265812 21512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "b70982305d8542b2aa0b02a1432868bb"
format_stamp: "Formatted at 2026-09-27 15:01:10 on dist-test-slave-pz0h"
server_key: "3517e6320b8a9a79102c22e4fec215cd"
server_key_iv: "9f8d24ad6346e91c03518e8a5c356876"
server_key_version: "encryptionkey@0"
I20260927 15:01:10.265930 21512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:01:10.270712 21512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:01:10.275058 21512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:01:10.275270 21512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:01:10.282364 21512 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40237
I20260927 15:01:10.282387 21579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40237 every 8 connection(s)
I20260927 15:01:10.283396 21512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:01:10.285600 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 21512
I20260927 15:01:10.285737 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 15:01:10.286083 392 external_mini_cluster.cc:1428] Setting key 1f3dcc1821a0b0533a0608ced4e83fe7
I20260927 15:01:10.286526 21580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:01:10.291559 21583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:10.287615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.153:58237 (local address 127.0.0.1:40237)
0927 15:01:10.288114 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:10.288129 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:10.288166 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:01:10.288839 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:10.288858 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:10.289156 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:10.289481 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:10.290296 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:10.290301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:10.290313 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:01:10.290324 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:10.290380 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:01:10.290400 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:10.290529 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:10.290637 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 15:01:10.290825 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":152,"threads_started":1}
I20260927 15:01:10.293527 21580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b70982305d8542b2aa0b02a1432868bb: Bootstrap starting.
I20260927 15:01:10.295324 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:01:10.296471 21580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b70982305d8542b2aa0b02a1432868bb: Neither blocks nor log segments found. Creating new log.
I20260927 15:01:10.297472 21580 log.cc:824] T 00000000000000000000000000000000 P b70982305d8542b2aa0b02a1432868bb: Log is configured to *not* fsync() on all Append() calls
I20260927 15:01:10.299429 21583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:10.297801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.153:58919 (local address 127.0.0.1:40237)
0927 15:01:10.297941 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:10.297946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:10.298083 (+ 137us) server_negotiation.cc:407] Connection header received
0927 15:01:10.298188 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:10.298191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:10.298240 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:10.298333 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:10.299055 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:10.299057 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:10.299059 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:01:10.299071 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:10.299098 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:01:10.299108 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:10.299173 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:10.299268 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 15:01:10.299305 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:01:10.299962 21580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b70982305d8542b2aa0b02a1432868bb: No bootstrap required, opened a new log
I20260927 15:01:10.303957 21580 raft_consensus.cc:348] T 00000000000000000000000000000000 P b70982305d8542b2aa0b02a1432868bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b70982305d8542b2aa0b02a1432868bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40237 } }
I20260927 15:01:10.304242 21580 raft_consensus.cc:374] T 00000000000000000000000000000000 P b70982305d8542b2aa0b02a1432868bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:01:10.304301 21580 raft_consensus.cc:729] T 00000000000000000000000000000000 P b70982305d8542b2aa0b02a1432868bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b70982305d8542b2aa0b02a1432868bb, State: Initialized, Role: FOLLOWER
I20260927 15:01:10.304746 21580 consensus_queue.cc:260] T 00000000000000000000000000000000 P b70982305d8542b2aa0b02a1432868bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b70982305d8542b2aa0b02a1432868bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40237 } }
I20260927 15:01:10.304885 21580 raft_consensus.cc:388] T 00000000000000000000000000000000 P b70982305d8542b2aa0b02a1432868bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:01:10.304953 21580 raft_consensus.cc:482] T 00000000000000000000000000000000 P b70982305d8542b2aa0b02a1432868bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:01:10.305058 21580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b70982305d8542b2aa0b02a1432868bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:01:10.306082 21580 raft_consensus.cc:504] T 00000000000000000000000000000000 P b70982305d8542b2aa0b02a1432868bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b70982305d8542b2aa0b02a1432868bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40237 } }
I20260927 15:01:10.306449 21580 leader_election.cc:302] T 00000000000000000000000000000000 P b70982305d8542b2aa0b02a1432868bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b70982305d8542b2aa0b02a1432868bb; no voters:
I20260927 15:01:10.306859 21580 leader_election.cc:288] T 00000000000000000000000000000000 P b70982305d8542b2aa0b02a1432868bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:01:10.306981 21590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b70982305d8542b2aa0b02a1432868bb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:01:10.307221 21590 raft_consensus.cc:686] T 00000000000000000000000000000000 P b70982305d8542b2aa0b02a1432868bb [term 1 LEADER]: Becoming Leader. State: Replica: b70982305d8542b2aa0b02a1432868bb, State: Running, Role: LEADER
I20260927 15:01:10.307539 21590 consensus_queue.cc:237] T 00000000000000000000000000000000 P b70982305d8542b2aa0b02a1432868bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b70982305d8542b2aa0b02a1432868bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40237 } }
I20260927 15:01:10.308264 21580 sys_catalog.cc:564] T 00000000000000000000000000000000 P b70982305d8542b2aa0b02a1432868bb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:01:10.309427 21591 sys_catalog.cc:455] T 00000000000000000000000000000000 P b70982305d8542b2aa0b02a1432868bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b70982305d8542b2aa0b02a1432868bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b70982305d8542b2aa0b02a1432868bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40237 } } }
I20260927 15:01:10.309574 21591 sys_catalog.cc:458] T 00000000000000000000000000000000 P b70982305d8542b2aa0b02a1432868bb [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:10.309908 21592 sys_catalog.cc:455] T 00000000000000000000000000000000 P b70982305d8542b2aa0b02a1432868bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b70982305d8542b2aa0b02a1432868bb. Latest consensus state: current_term: 1 leader_uuid: "b70982305d8542b2aa0b02a1432868bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b70982305d8542b2aa0b02a1432868bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40237 } } }
I20260927 15:01:10.310016 21592 sys_catalog.cc:458] T 00000000000000000000000000000000 P b70982305d8542b2aa0b02a1432868bb [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:10.310117 21599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:01:10.313274 21599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:01:10.319485 21599 catalog_manager.cc:1347] Generated new cluster ID: f6da35c1fff1400a8cce1979ee92f588
I20260927 15:01:10.319588 21599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:01:10.336300 21599 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:01:10.336666 21599 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:01:10.343312 21599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b70982305d8542b2aa0b02a1432868bb: Generated new TSK 0
I20260927 15:01:10.344260 21599 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:01:10.359226 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 21512
2026-09-27T15:01:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:01:10 dist-test-slave-pz0h krb5kdc[21611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:01:10 dist-test-slave-pz0h krb5kdc[21611](info): set up 2 sockets
Sep 27 15:01:10 dist-test-slave-pz0h krb5kdc[21611](info): commencing operation
krb5kdc: starting...
W20260927 15:01:11.306838 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 15:01:11 dist-test-slave-pz0h krb5kdc[21611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:01:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:01:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:01:11.426784 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43483
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:01:11.538491 21624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:01:11.538822 21624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:01:11.538887 21624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:01:11.542980 21624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:01:11.543056 21624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:01:11.543082 21624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:01:11.543105 21624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:01:11.543126 21624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:01:11.548683 21624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43483
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:01:11.549808 21624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:01:11.550882 21624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:01:11.559321 21632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:11.559324 21629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:11.559321 21630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:01:11.559674 21624 server_base.cc:1034] running on GCE node
I20260927 15:01:11.560240 21624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:01:11.561400 21624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:01:11.562561 21624 hybrid_clock.cc:648] HybridClock initialized: now 1790521271562550 us; error 33 us; skew 500 ppm
Sep 27 15:01:11 dist-test-slave-pz0h krb5kdc[21611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:01:11.565687 21624 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:01:11.566716 21624 webserver.cc:460] Webserver started at http://127.0.0.1:40843/ using document root <none> and password file <none>
I20260927 15:01:11.567299 21624 fs_manager.cc:359] Metadata directory not provided
I20260927 15:01:11.567363 21624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:01:11.567541 21624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:01:11.569626 21624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "4b432b88a79f4a96879c523afff01d10"
format_stamp: "Formatted at 2026-09-27 15:01:11 on dist-test-slave-pz0h"
server_key: "4494ee07f69bdfe0f367e302616fe579"
server_key_iv: "3cb068b517fb7620ae68340f3812ceda"
server_key_version: "encryptionkey@0"
I20260927 15:01:11.570166 21624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "4b432b88a79f4a96879c523afff01d10"
format_stamp: "Formatted at 2026-09-27 15:01:11 on dist-test-slave-pz0h"
server_key: "4494ee07f69bdfe0f367e302616fe579"
server_key_iv: "3cb068b517fb7620ae68340f3812ceda"
server_key_version: "encryptionkey@0"
I20260927 15:01:11.574234 21624 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:01:11.576933 21639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:01:11.578263 21624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:01:11.578393 21624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "4b432b88a79f4a96879c523afff01d10"
format_stamp: "Formatted at 2026-09-27 15:01:11 on dist-test-slave-pz0h"
server_key: "4494ee07f69bdfe0f367e302616fe579"
server_key_iv: "3cb068b517fb7620ae68340f3812ceda"
server_key_version: "encryptionkey@0"
I20260927 15:01:11.578527 21624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:01:11.601434 21624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:01:11.604784 21624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:01:11.604969 21624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:01:11.612334 21624 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43483
I20260927 15:01:11.612344 21691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43483 every 8 connection(s)
I20260927 15:01:11.613363 21624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:01:11.613615 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 21624
I20260927 15:01:11.613741 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 15:01:11.614055 392 external_mini_cluster.cc:1428] Setting key 6ebec42ddcb1f5cad94dc9284b45cf53
I20260927 15:01:11.616384 21692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:01:11.622499 21692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b432b88a79f4a96879c523afff01d10: Bootstrap starting.
Sep 27 15:01:11 dist-test-slave-pz0h krb5kdc[21611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 15:01:11.625196 21692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b432b88a79f4a96879c523afff01d10: Neither blocks nor log segments found. Creating new log.
I20260927 15:01:11.625972 21692 log.cc:824] T 00000000000000000000000000000000 P 4b432b88a79f4a96879c523afff01d10: Log is configured to *not* fsync() on all Append() calls
I20260927 15:01:11.628130 21692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b432b88a79f4a96879c523afff01d10: No bootstrap required, opened a new log
I20260927 15:01:11.628712 21695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:11.615467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.153:42583 (local address 127.0.0.1:43483)
0927 15:01:11.615887 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:11.615903 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:11.615940 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:01:11.616678 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:11.616707 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:11.617100 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:11.617452 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:11.618287 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:11.619397 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:11.620268 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:11.620764 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:11.620911 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:11.623717 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:11.623722 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:11.623737 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:11.623772 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:11.625455 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:11.626023 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:11.626029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:11.626033 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:11.626184 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:11.626510 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:11.626513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:11.626514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:11.626978 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:11.627199 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:11.627478 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 15:01:11.627709 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":120,"threads_started":1}
I20260927 15:01:11.632058 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:01:11.631943 21692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b432b88a79f4a96879c523afff01d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b432b88a79f4a96879c523afff01d10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43483 } }
I20260927 15:01:11.632233 21692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b432b88a79f4a96879c523afff01d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:01:11.632292 21692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b432b88a79f4a96879c523afff01d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b432b88a79f4a96879c523afff01d10, State: Initialized, Role: FOLLOWER
I20260927 15:01:11.632709 21692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b432b88a79f4a96879c523afff01d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b432b88a79f4a96879c523afff01d10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43483 } }
I20260927 15:01:11.632828 21692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b432b88a79f4a96879c523afff01d10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:01:11.632885 21692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b432b88a79f4a96879c523afff01d10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:01:11.632977 21692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b432b88a79f4a96879c523afff01d10 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:01:11.634018 21692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b432b88a79f4a96879c523afff01d10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b432b88a79f4a96879c523afff01d10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43483 } }
I20260927 15:01:11.634402 21692 leader_election.cc:302] T 00000000000000000000000000000000 P 4b432b88a79f4a96879c523afff01d10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b432b88a79f4a96879c523afff01d10; no voters:
I20260927 15:01:11.634781 21692 leader_election.cc:288] T 00000000000000000000000000000000 P 4b432b88a79f4a96879c523afff01d10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:01:11.634912 21702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b432b88a79f4a96879c523afff01d10 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:01:11.635113 21702 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b432b88a79f4a96879c523afff01d10 [term 1 LEADER]: Becoming Leader. State: Replica: 4b432b88a79f4a96879c523afff01d10, State: Running, Role: LEADER
I20260927 15:01:11.635438 21702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b432b88a79f4a96879c523afff01d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b432b88a79f4a96879c523afff01d10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43483 } }
I20260927 15:01:11.636078 21692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b432b88a79f4a96879c523afff01d10 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:01:11.637123 21703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b432b88a79f4a96879c523afff01d10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b432b88a79f4a96879c523afff01d10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b432b88a79f4a96879c523afff01d10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43483 } } }
I20260927 15:01:11.637257 21703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b432b88a79f4a96879c523afff01d10 [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:11.637518 21703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b432b88a79f4a96879c523afff01d10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b432b88a79f4a96879c523afff01d10. Latest consensus state: current_term: 1 leader_uuid: "4b432b88a79f4a96879c523afff01d10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b432b88a79f4a96879c523afff01d10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43483 } } }
I20260927 15:01:11.637596 21703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b432b88a79f4a96879c523afff01d10 [sys.catalog]: This master's current role is: LEADER
W20260927 15:01:11.640920 21717 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4b432b88a79f4a96879c523afff01d10: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:01:11.641004 21717 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:01:11.641070 21711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:01:11.641928 21711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:01:11.642872 21695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:11.634128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.153:33325 (local address 127.0.0.1:43483)
0927 15:01:11.634246 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:11.634250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:11.634380 (+ 130us) server_negotiation.cc:407] Connection header received
0927 15:01:11.634516 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:11.634519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:11.634573 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:11.634658 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:11.635438 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:11.636230 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:11.637358 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:11.637710 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:11.637813 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:11.639534 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:11.639538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:11.639540 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:11.639566 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:11.640974 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:11.641562 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:11.641565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:11.641566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:11.641645 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:11.641967 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:11.641969 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:11.641970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:11.642168 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:11.642297 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:11.642598 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 15:01:11.642696 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:01:11.647704 21711 catalog_manager.cc:1347] Generated new cluster ID: 02bc084159f142e39b328db32904a8ee
I20260927 15:01:11.647801 21711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:01:11.668893 21711 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:01:11.670011 21711 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:01:11.674957 21711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b432b88a79f4a96879c523afff01d10: Generated new TSK 0
I20260927 15:01:11.675907 21711 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:01:11.681520 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 21624
W20260927 15:01:11.686363 21609 connection.cc:535] client connection to 127.0.0.1:43483 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43483): Connection reset by peer (error 104)
2026-09-27T15:01:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:01:11 dist-test-slave-pz0h krb5kdc[21723](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:01:11 dist-test-slave-pz0h krb5kdc[21723](info): set up 2 sockets
Sep 27 15:01:11 dist-test-slave-pz0h krb5kdc[21723](info): commencing operation
krb5kdc: starting...
W20260927 15:01:12.626783 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 15:01:12 dist-test-slave-pz0h krb5kdc[21723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:01:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:01:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:01:12.745671 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40623
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:01:12.858443 21736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:01:12.858764 21736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:01:12.858852 21736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:01:12.863057 21736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:01:12.863138 21736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:01:12.863157 21736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:01:12.863173 21736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:01:12.863188 21736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:01:12.868871 21736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40623
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:01:12.870048 21736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:01:12.871120 21736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:01:12.879995 21741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:12.880143 21742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:12.880113 21744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:01:12.880533 21736 server_base.cc:1034] running on GCE node
I20260927 15:01:12.881172 21736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:01:12.882406 21736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:01:12.883595 21736 hybrid_clock.cc:648] HybridClock initialized: now 1790521272883564 us; error 49 us; skew 500 ppm
Sep 27 15:01:12 dist-test-slave-pz0h krb5kdc[21723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:01:12.886832 21736 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:01:12.888027 21736 webserver.cc:460] Webserver started at http://127.0.0.1:40165/ using document root <none> and password file <none>
I20260927 15:01:12.888629 21736 fs_manager.cc:359] Metadata directory not provided
I20260927 15:01:12.888707 21736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:01:12.888921 21736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:01:12.891209 21736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "17e6f78f988a4546a710d8439b3946b0"
format_stamp: "Formatted at 2026-09-27 15:01:12 on dist-test-slave-pz0h"
server_key: "0780e9a672306c554d7b873ff07de959"
server_key_iv: "249d7163bfc9a70b53bf47d6d991aac1"
server_key_version: "encryptionkey@0"
I20260927 15:01:12.891790 21736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "17e6f78f988a4546a710d8439b3946b0"
format_stamp: "Formatted at 2026-09-27 15:01:12 on dist-test-slave-pz0h"
server_key: "0780e9a672306c554d7b873ff07de959"
server_key_iv: "249d7163bfc9a70b53bf47d6d991aac1"
server_key_version: "encryptionkey@0"
I20260927 15:01:12.895831 21736 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:01:12.898607 21751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:01:12.900249 21736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:01:12.900431 21736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "17e6f78f988a4546a710d8439b3946b0"
format_stamp: "Formatted at 2026-09-27 15:01:12 on dist-test-slave-pz0h"
server_key: "0780e9a672306c554d7b873ff07de959"
server_key_iv: "249d7163bfc9a70b53bf47d6d991aac1"
server_key_version: "encryptionkey@0"
I20260927 15:01:12.900568 21736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:01:12.917553 21736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:01:12.920966 21736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:01:12.921175 21736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:01:12.928301 21736 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40623
I20260927 15:01:12.928332 21803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40623 every 8 connection(s)
I20260927 15:01:12.929284 21736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:01:12.932215 21804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:01:12.932226 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 21736
I20260927 15:01:12.932344 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 15:01:12.932637 392 external_mini_cluster.cc:1428] Setting key 2daac38c581a467f6751ad15da57c373
I20260927 15:01:12.939442 21804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17e6f78f988a4546a710d8439b3946b0: Bootstrap starting.
I20260927 15:01:12.941677 21807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:12.934052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.153:51223 (local address 127.0.0.1:40623)
0927 15:01:12.934493 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:12.934509 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:12.934546 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:01:12.935191 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:12.935209 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:12.935575 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:12.935860 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:12.936230 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:12.937264 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:12.938404 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:12.938937 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:12.939084 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:12.939988 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:12.939994 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:12.940007 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:01:12.940407 (+ 400us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:12.940465 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:01:12.940489 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:12.940605 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:12.940788 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 15:01:12.940930 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":127,"threads_started":1}
I20260927 15:01:12.941823 21804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17e6f78f988a4546a710d8439b3946b0: Neither blocks nor log segments found. Creating new log.
I20260927 15:01:12.942556 21804 log.cc:824] T 00000000000000000000000000000000 P 17e6f78f988a4546a710d8439b3946b0: Log is configured to *not* fsync() on all Append() calls
I20260927 15:01:12.944631 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:01:12.944769 21804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17e6f78f988a4546a710d8439b3946b0: No bootstrap required, opened a new log
I20260927 15:01:12.949138 21804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17e6f78f988a4546a710d8439b3946b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e6f78f988a4546a710d8439b3946b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40623 } }
I20260927 15:01:12.949533 21804 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17e6f78f988a4546a710d8439b3946b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:01:12.949591 21804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17e6f78f988a4546a710d8439b3946b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17e6f78f988a4546a710d8439b3946b0, State: Initialized, Role: FOLLOWER
I20260927 15:01:12.950135 21804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17e6f78f988a4546a710d8439b3946b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e6f78f988a4546a710d8439b3946b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40623 } }
I20260927 15:01:12.950325 21804 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17e6f78f988a4546a710d8439b3946b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:01:12.950408 21804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17e6f78f988a4546a710d8439b3946b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:01:12.950531 21804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17e6f78f988a4546a710d8439b3946b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:01:12.950881 21807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:12.946481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.153:39209 (local address 127.0.0.1:40623)
0927 15:01:12.946615 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:12.946619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:12.946783 (+ 164us) server_negotiation.cc:407] Connection header received
0927 15:01:12.946915 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:12.946919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:12.946974 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:12.947066 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:12.947273 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:12.948050 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:12.948923 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:12.949254 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:12.949365 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:12.950271 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:12.950275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:12.950278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:01:12.950295 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:12.950328 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:01:12.950342 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:12.950432 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:12.950654 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 15:01:12.950700 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:01:12.951858 21804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17e6f78f988a4546a710d8439b3946b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e6f78f988a4546a710d8439b3946b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40623 } }
I20260927 15:01:12.952297 21804 leader_election.cc:302] T 00000000000000000000000000000000 P 17e6f78f988a4546a710d8439b3946b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17e6f78f988a4546a710d8439b3946b0; no voters:
I20260927 15:01:12.952662 21804 leader_election.cc:288] T 00000000000000000000000000000000 P 17e6f78f988a4546a710d8439b3946b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:01:12.952791 21814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17e6f78f988a4546a710d8439b3946b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:01:12.953013 21814 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17e6f78f988a4546a710d8439b3946b0 [term 1 LEADER]: Becoming Leader. State: Replica: 17e6f78f988a4546a710d8439b3946b0, State: Running, Role: LEADER
I20260927 15:01:12.953387 21814 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17e6f78f988a4546a710d8439b3946b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e6f78f988a4546a710d8439b3946b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40623 } }
I20260927 15:01:12.954000 21804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17e6f78f988a4546a710d8439b3946b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:01:12.955009 21815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17e6f78f988a4546a710d8439b3946b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17e6f78f988a4546a710d8439b3946b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e6f78f988a4546a710d8439b3946b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40623 } } }
I20260927 15:01:12.955127 21815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17e6f78f988a4546a710d8439b3946b0 [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:12.955425 21816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17e6f78f988a4546a710d8439b3946b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17e6f78f988a4546a710d8439b3946b0. Latest consensus state: current_term: 1 leader_uuid: "17e6f78f988a4546a710d8439b3946b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e6f78f988a4546a710d8439b3946b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40623 } } }
I20260927 15:01:12.955506 21823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:01:12.955582 21816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17e6f78f988a4546a710d8439b3946b0 [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:12.958567 21823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:01:12.965039 21823 catalog_manager.cc:1347] Generated new cluster ID: 5ee7a5d32b4d408894a7eb7ff166ba85
I20260927 15:01:12.965194 21823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:01:12.983486 21823 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:01:12.984496 21823 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:01:12.990753 21823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17e6f78f988a4546a710d8439b3946b0: Generated new TSK 0
I20260927 15:01:12.991530 21823 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:01:12.998713 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 21736
W20260927 15:01:13.003602 21721 connection.cc:535] client connection to 127.0.0.1:40623 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40623): Connection reset by peer (error 104)
2026-09-27T15:01:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:01:13 dist-test-slave-pz0h krb5kdc[21835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:01:13 dist-test-slave-pz0h krb5kdc[21835](info): set up 2 sockets
Sep 27 15:01:13 dist-test-slave-pz0h krb5kdc[21835](info): commencing operation
krb5kdc: starting...
W20260927 15:01:13.955338 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 15:01:14 dist-test-slave-pz0h krb5kdc[21835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:01:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:01:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:01:14.078001 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45849
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:01:14.191505 21848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:01:14.191815 21848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:01:14.191861 21848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:01:14.196090 21848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:01:14.196178 21848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:01:14.196197 21848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:01:14.196214 21848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:01:14.196228 21848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:01:14.202080 21848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45849
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521004237705-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:01:14.203274 21848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:01:14.204339 21848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:01:14.212911 21853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:14.212890 21854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:14.212936 21856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:01:14.213114 21848 server_base.cc:1034] running on GCE node
I20260927 15:01:14.213892 21848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:01:14.215152 21848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:01:14.216317 21848 hybrid_clock.cc:648] HybridClock initialized: now 1790521274216304 us; error 34 us; skew 500 ppm
Sep 27 15:01:14 dist-test-slave-pz0h krb5kdc[21835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:01:14.224759 21848 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:01:14.225849 21848 webserver.cc:460] Webserver started at http://127.0.0.1:45939/ using document root <none> and password file <none>
I20260927 15:01:14.226429 21848 fs_manager.cc:359] Metadata directory not provided
I20260927 15:01:14.226504 21848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:01:14.226715 21848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:01:14.228971 21848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "242e1140b96144fe91996bfffdc9e468"
format_stamp: "Formatted at 2026-09-27 15:01:14 on dist-test-slave-pz0h"
server_key: "30e3c7da364caba917f821d875de5926"
server_key_iv: "84e59108d7d65e5965d41f2e326aef5c"
server_key_version: "encryptionkey@0"
I20260927 15:01:14.229574 21848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "242e1140b96144fe91996bfffdc9e468"
format_stamp: "Formatted at 2026-09-27 15:01:14 on dist-test-slave-pz0h"
server_key: "30e3c7da364caba917f821d875de5926"
server_key_iv: "84e59108d7d65e5965d41f2e326aef5c"
server_key_version: "encryptionkey@0"
I20260927 15:01:14.233708 21848 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:01:14.237614 21863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:01:14.239434 21848 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:01:14.239570 21848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "242e1140b96144fe91996bfffdc9e468"
format_stamp: "Formatted at 2026-09-27 15:01:14 on dist-test-slave-pz0h"
server_key: "30e3c7da364caba917f821d875de5926"
server_key_iv: "84e59108d7d65e5965d41f2e326aef5c"
server_key_version: "encryptionkey@0"
I20260927 15:01:14.239688 21848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:01:14.244611 21848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:01:14.247892 21848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:01:14.248090 21848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:01:14.255409 21848 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45849
I20260927 15:01:14.255417 21915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45849 every 8 connection(s)
I20260927 15:01:14.256481 21848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:01:14.259567 21916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:01:14.264920 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 21848
I20260927 15:01:14.265077 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 15:01:14.265348 392 external_mini_cluster.cc:1428] Setting key 1ac9edf01c6681833dd20bf25ff4730c
I20260927 15:01:14.265710 21916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 242e1140b96144fe91996bfffdc9e468: Bootstrap starting.
I20260927 15:01:14.268360 21916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 242e1140b96144fe91996bfffdc9e468: Neither blocks nor log segments found. Creating new log.
I20260927 15:01:14.269275 21916 log.cc:824] T 00000000000000000000000000000000 P 242e1140b96144fe91996bfffdc9e468: Log is configured to *not* fsync() on all Append() calls
I20260927 15:01:14.271066 21919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:14.266761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.153:49257 (local address 127.0.0.1:45849)
0927 15:01:14.267249 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:14.267264 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:14.267304 (+ 40us) server_negotiation.cc:407] Connection header received
0927 15:01:14.268026 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:14.268045 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:14.268369 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:14.268732 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:14.269569 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:14.269573 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:14.269586 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:01:14.269598 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:14.269656 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:01:14.269676 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:14.269781 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:14.269903 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 15:01:14.270079 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":119,"threads_started":1}
I20260927 15:01:14.271723 21916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 242e1140b96144fe91996bfffdc9e468: No bootstrap required, opened a new log
I20260927 15:01:14.274783 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:01:14.276055 21916 raft_consensus.cc:348] T 00000000000000000000000000000000 P 242e1140b96144fe91996bfffdc9e468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "242e1140b96144fe91996bfffdc9e468" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45849 } }
I20260927 15:01:14.276343 21916 raft_consensus.cc:374] T 00000000000000000000000000000000 P 242e1140b96144fe91996bfffdc9e468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:01:14.276379 21916 raft_consensus.cc:729] T 00000000000000000000000000000000 P 242e1140b96144fe91996bfffdc9e468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 242e1140b96144fe91996bfffdc9e468, State: Initialized, Role: FOLLOWER
I20260927 15:01:14.276850 21916 consensus_queue.cc:260] T 00000000000000000000000000000000 P 242e1140b96144fe91996bfffdc9e468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "242e1140b96144fe91996bfffdc9e468" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45849 } }
I20260927 15:01:14.277020 21916 raft_consensus.cc:388] T 00000000000000000000000000000000 P 242e1140b96144fe91996bfffdc9e468 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:01:14.277096 21916 raft_consensus.cc:482] T 00000000000000000000000000000000 P 242e1140b96144fe91996bfffdc9e468 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:01:14.277222 21916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 242e1140b96144fe91996bfffdc9e468 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:01:14.278227 21916 raft_consensus.cc:504] T 00000000000000000000000000000000 P 242e1140b96144fe91996bfffdc9e468 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "242e1140b96144fe91996bfffdc9e468" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45849 } }
I20260927 15:01:14.278509 21919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:14.276885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.153:51751 (local address 127.0.0.1:45849)
0927 15:01:14.277030 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:14.277035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:14.277186 (+ 151us) server_negotiation.cc:407] Connection header received
0927 15:01:14.277344 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:14.277348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:14.277387 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:14.277454 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:14.278139 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:14.278143 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:14.278145 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:01:14.278157 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:14.278185 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:01:14.278196 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:14.278278 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:14.278341 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 15:01:14.278383 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:01:14.278640 21916 leader_election.cc:302] T 00000000000000000000000000000000 P 242e1140b96144fe91996bfffdc9e468 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 242e1140b96144fe91996bfffdc9e468; no voters:
I20260927 15:01:14.279004 21916 leader_election.cc:288] T 00000000000000000000000000000000 P 242e1140b96144fe91996bfffdc9e468 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:01:14.279098 21926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 242e1140b96144fe91996bfffdc9e468 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:01:14.279295 21926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 242e1140b96144fe91996bfffdc9e468 [term 1 LEADER]: Becoming Leader. State: Replica: 242e1140b96144fe91996bfffdc9e468, State: Running, Role: LEADER
I20260927 15:01:14.279629 21926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 242e1140b96144fe91996bfffdc9e468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "242e1140b96144fe91996bfffdc9e468" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45849 } }
I20260927 15:01:14.280309 21916 sys_catalog.cc:564] T 00000000000000000000000000000000 P 242e1140b96144fe91996bfffdc9e468 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:01:14.281383 21928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 242e1140b96144fe91996bfffdc9e468 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 242e1140b96144fe91996bfffdc9e468. Latest consensus state: current_term: 1 leader_uuid: "242e1140b96144fe91996bfffdc9e468" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "242e1140b96144fe91996bfffdc9e468" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45849 } } }
I20260927 15:01:14.281526 21928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 242e1140b96144fe91996bfffdc9e468 [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:14.281387 21927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 242e1140b96144fe91996bfffdc9e468 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "242e1140b96144fe91996bfffdc9e468" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "242e1140b96144fe91996bfffdc9e468" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45849 } } }
I20260927 15:01:14.281682 21927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 242e1140b96144fe91996bfffdc9e468 [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:14.281945 21935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:01:14.285749 21935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:01:14.292899 21935 catalog_manager.cc:1347] Generated new cluster ID: e0ce4bebab094d87a44a8fc099b1f259
I20260927 15:01:14.293004 21935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:01:14.316870 21935 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:01:14.317219 21935 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:01:14.333401 21935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 242e1140b96144fe91996bfffdc9e468: Generated new TSK 0
I20260927 15:01:14.334255 21935 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:01:14.340950 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 21848
2026-09-27T15:01:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:01:14 dist-test-slave-pz0h krb5kdc[21947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:01:14 dist-test-slave-pz0h krb5kdc[21947](info): set up 2 sockets
Sep 27 15:01:14 dist-test-slave-pz0h krb5kdc[21947](info): commencing operation
krb5kdc: starting...
W20260927 15:01:15.290943 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 15:01:15 dist-test-slave-pz0h krb5kdc[21947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:01:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:01:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:01:15.414098 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40359
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:01:15.526731 21960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:01:15.527057 21960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:01:15.527120 21960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:01:15.531286 21960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:01:15.531361 21960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:01:15.531386 21960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:01:15.531411 21960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:01:15.531440 21960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:01:15.537389 21960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40359
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:01:15.538579 21960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:01:15.539677 21960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:01:15.548071 21966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:15.548094 21965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:01:15.548179 21960 server_base.cc:1034] running on GCE node
W20260927 15:01:15.548075 21968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:01:15.548934 21960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:01:15.550076 21960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:01:15.551223 21960 hybrid_clock.cc:648] HybridClock initialized: now 1790521275551210 us; error 32 us; skew 500 ppm
Sep 27 15:01:15 dist-test-slave-pz0h krb5kdc[21947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:01:15.553895 21960 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:01:15.554836 21960 webserver.cc:460] Webserver started at http://127.0.98.62:42577/ using document root <none> and password file <none>
I20260927 15:01:15.555389 21960 fs_manager.cc:359] Metadata directory not provided
I20260927 15:01:15.555449 21960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:01:15.555655 21960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:01:15.557760 21960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "b8c32e7b69684fa0a7cddd6e7e68351e"
format_stamp: "Formatted at 2026-09-27 15:01:15 on dist-test-slave-pz0h"
server_key: "5d69ae2cd27d67afa5cec29ef77b830e"
server_key_iv: "dfaaaba34ac2c6796aa7b96a64680a7a"
server_key_version: "encryptionkey@0"
I20260927 15:01:15.558353 21960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "b8c32e7b69684fa0a7cddd6e7e68351e"
format_stamp: "Formatted at 2026-09-27 15:01:15 on dist-test-slave-pz0h"
server_key: "5d69ae2cd27d67afa5cec29ef77b830e"
server_key_iv: "dfaaaba34ac2c6796aa7b96a64680a7a"
server_key_version: "encryptionkey@0"
I20260927 15:01:15.562328 21960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:01:15.564965 21975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:01:15.566331 21960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:01:15.566473 21960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "b8c32e7b69684fa0a7cddd6e7e68351e"
format_stamp: "Formatted at 2026-09-27 15:01:15 on dist-test-slave-pz0h"
server_key: "5d69ae2cd27d67afa5cec29ef77b830e"
server_key_iv: "dfaaaba34ac2c6796aa7b96a64680a7a"
server_key_version: "encryptionkey@0"
I20260927 15:01:15.566597 21960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:01:15.580530 21960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:01:15.585139 21960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:01:15.585373 21960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:01:15.592831 21960 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40359
I20260927 15:01:15.592855 22027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40359 every 8 connection(s)
I20260927 15:01:15.593878 21960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:01:15.597076 22028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:01:15.600839 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 21960
I20260927 15:01:15.600980 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 15:01:15.601225 392 external_mini_cluster.cc:1428] Setting key 77438406f8574d858fe4e8b4dd51a924
I20260927 15:01:15.603514 22028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8c32e7b69684fa0a7cddd6e7e68351e: Bootstrap starting.
I20260927 15:01:15.606451 22028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8c32e7b69684fa0a7cddd6e7e68351e: Neither blocks nor log segments found. Creating new log.
I20260927 15:01:15.607479 22028 log.cc:824] T 00000000000000000000000000000000 P b8c32e7b69684fa0a7cddd6e7e68351e: Log is configured to *not* fsync() on all Append() calls
I20260927 15:01:15.610055 22028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8c32e7b69684fa0a7cddd6e7e68351e: No bootstrap required, opened a new log
Sep 27 15:01:15 dist-test-slave-pz0h krb5kdc[21947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:01:15.614192 22028 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8c32e7b69684fa0a7cddd6e7e68351e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c32e7b69684fa0a7cddd6e7e68351e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40359 } }
I20260927 15:01:15.614486 22028 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8c32e7b69684fa0a7cddd6e7e68351e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:01:15.614549 22028 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8c32e7b69684fa0a7cddd6e7e68351e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8c32e7b69684fa0a7cddd6e7e68351e, State: Initialized, Role: FOLLOWER
I20260927 15:01:15.615088 22028 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8c32e7b69684fa0a7cddd6e7e68351e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c32e7b69684fa0a7cddd6e7e68351e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40359 } }
I20260927 15:01:15.615253 22028 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8c32e7b69684fa0a7cddd6e7e68351e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:01:15.615337 22028 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8c32e7b69684fa0a7cddd6e7e68351e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:01:15.615453 22028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8c32e7b69684fa0a7cddd6e7e68351e [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:01:15.616681 22028 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8c32e7b69684fa0a7cddd6e7e68351e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c32e7b69684fa0a7cddd6e7e68351e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40359 } }
I20260927 15:01:15.617014 22028 leader_election.cc:302] T 00000000000000000000000000000000 P b8c32e7b69684fa0a7cddd6e7e68351e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8c32e7b69684fa0a7cddd6e7e68351e; no voters:
I20260927 15:01:15.617349 22028 leader_election.cc:288] T 00000000000000000000000000000000 P b8c32e7b69684fa0a7cddd6e7e68351e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:01:15.617436 22033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8c32e7b69684fa0a7cddd6e7e68351e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:01:15.617575 22031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:15.602692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36315 (local address 127.0.98.62:40359)
0927 15:01:15.603226 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:15.603246 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:15.603291 (+ 45us) server_negotiation.cc:407] Connection header received
0927 15:01:15.604179 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:15.604206 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:15.604683 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:15.605120 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:15.605893 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:15.607463 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:15.608683 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:15.609399 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:15.609585 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:15.612543 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:15.612549 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:15.612566 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:15.612613 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:15.614282 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:15.614956 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:15.614961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:15.614966 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:15.615090 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:15.615427 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:15.615430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:15.615432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:15.615904 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:15.616085 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:15.616401 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 15:01:15.616651 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":132,"threads_started":1}
I20260927 15:01:15.618547 22028 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8c32e7b69684fa0a7cddd6e7e68351e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:01:15.617599 22033 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8c32e7b69684fa0a7cddd6e7e68351e [term 1 LEADER]: Becoming Leader. State: Replica: b8c32e7b69684fa0a7cddd6e7e68351e, State: Running, Role: LEADER
I20260927 15:01:15.619040 22033 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8c32e7b69684fa0a7cddd6e7e68351e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c32e7b69684fa0a7cddd6e7e68351e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40359 } }
I20260927 15:01:15.620582 22035 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8c32e7b69684fa0a7cddd6e7e68351e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8c32e7b69684fa0a7cddd6e7e68351e. Latest consensus state: current_term: 1 leader_uuid: "b8c32e7b69684fa0a7cddd6e7e68351e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c32e7b69684fa0a7cddd6e7e68351e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40359 } } }
I20260927 15:01:15.620713 22035 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8c32e7b69684fa0a7cddd6e7e68351e [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:15.621033 22034 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8c32e7b69684fa0a7cddd6e7e68351e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8c32e7b69684fa0a7cddd6e7e68351e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c32e7b69684fa0a7cddd6e7e68351e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40359 } } }
I20260927 15:01:15.621124 22034 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8c32e7b69684fa0a7cddd6e7e68351e [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:15.621232 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 15:01:15.623725 22047 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b8c32e7b69684fa0a7cddd6e7e68351e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:01:15.623816 22047 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:01:15.623878 22048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:01:15.624773 22048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:01:15.631372 22048 catalog_manager.cc:1347] Generated new cluster ID: 232395a97aef42028e3fc6c74ba6ff1a
I20260927 15:01:15.631453 22048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:01:15.632503 22031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:15.623295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48033 (local address 127.0.98.62:40359)
0927 15:01:15.623394 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:15.623399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:15.623877 (+ 478us) server_negotiation.cc:407] Connection header received
0927 15:01:15.623931 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:15.623934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:15.623982 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:15.624064 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:15.624917 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:15.625708 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:15.626833 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:15.627056 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:15.627148 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:15.628528 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:15.628531 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:15.628533 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:15.628558 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:15.630332 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:15.630919 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:15.630923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:15.630925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:15.631021 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:15.631406 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:15.631410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:15.631412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:15.631637 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:15.631778 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:15.632220 (+ 442us) server_negotiation.cc:299] Negotiation successful
0927 15:01:15.632336 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 15:01:15.657684 22048 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:01:15.659102 22048 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:01:15.667030 22048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8c32e7b69684fa0a7cddd6e7e68351e: Generated new TSK 0
I20260927 15:01:15.667964 22048 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:01:15.677306 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 21960
W20260927 15:01:15.682113 21945 connection.cc:535] client connection to 127.0.98.62:40359 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:40359): Connection reset by peer (error 104)
2026-09-27T15:01:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:01:15 dist-test-slave-pz0h krb5kdc[22059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:01:15 dist-test-slave-pz0h krb5kdc[22059](info): set up 2 sockets
Sep 27 15:01:15 dist-test-slave-pz0h krb5kdc[22059](info): commencing operation
krb5kdc: starting...
W20260927 15:01:16.631273 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 15:01:16 dist-test-slave-pz0h krb5kdc[22059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:01:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:01:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:01:16.753790 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43669
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:01:16.868574 22072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:01:16.868894 22072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:01:16.868953 22072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:01:16.873605 22072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:01:16.873721 22072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:01:16.873754 22072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:01:16.873775 22072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:01:16.873797 22072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:01:16.879690 22072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43669
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:01:16.880882 22072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:01:16.882107 22072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:01:16.891008 22077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:16.891018 22080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:16.891017 22078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:01:16.891247 22072 server_base.cc:1034] running on GCE node
I20260927 15:01:16.891980 22072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:01:16.893146 22072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:01:16.894342 22072 hybrid_clock.cc:648] HybridClock initialized: now 1790521276894322 us; error 34 us; skew 500 ppm
Sep 27 15:01:16 dist-test-slave-pz0h krb5kdc[22059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:01:16.897315 22072 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:01:16.898316 22072 webserver.cc:460] Webserver started at http://127.0.98.62:38443/ using document root <none> and password file <none>
I20260927 15:01:16.898963 22072 fs_manager.cc:359] Metadata directory not provided
I20260927 15:01:16.899036 22072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:01:16.899227 22072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:01:16.901404 22072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "cdd6563ef09445a381cb7a01837863fc"
format_stamp: "Formatted at 2026-09-27 15:01:16 on dist-test-slave-pz0h"
server_key: "e0d4b8e3da403ff35b0339427f45c356"
server_key_iv: "e02fedd875344a259f579859698dc785"
server_key_version: "encryptionkey@0"
I20260927 15:01:16.901948 22072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "cdd6563ef09445a381cb7a01837863fc"
format_stamp: "Formatted at 2026-09-27 15:01:16 on dist-test-slave-pz0h"
server_key: "e0d4b8e3da403ff35b0339427f45c356"
server_key_iv: "e02fedd875344a259f579859698dc785"
server_key_version: "encryptionkey@0"
I20260927 15:01:16.905874 22072 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 15:01:16.908550 22087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:01:16.910064 22072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:01:16.910203 22072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "cdd6563ef09445a381cb7a01837863fc"
format_stamp: "Formatted at 2026-09-27 15:01:16 on dist-test-slave-pz0h"
server_key: "e0d4b8e3da403ff35b0339427f45c356"
server_key_iv: "e02fedd875344a259f579859698dc785"
server_key_version: "encryptionkey@0"
I20260927 15:01:16.910331 22072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:01:16.921638 22072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:01:16.924728 22072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:01:16.924893 22072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:01:16.931704 22072 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43669
I20260927 15:01:16.931717 22139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43669 every 8 connection(s)
I20260927 15:01:16.932685 22072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:01:16.935439 22140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:01:16.940620 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 22072
I20260927 15:01:16.940763 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 15:01:16.940996 392 external_mini_cluster.cc:1428] Setting key cafe92c9f06a15d971291368556fe97c
I20260927 15:01:16.941221 22140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdd6563ef09445a381cb7a01837863fc: Bootstrap starting.
I20260927 15:01:16.944002 22140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdd6563ef09445a381cb7a01837863fc: Neither blocks nor log segments found. Creating new log.
I20260927 15:01:16.944926 22140 log.cc:824] T 00000000000000000000000000000000 P cdd6563ef09445a381cb7a01837863fc: Log is configured to *not* fsync() on all Append() calls
I20260927 15:01:16.947178 22140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdd6563ef09445a381cb7a01837863fc: No bootstrap required, opened a new log
I20260927 15:01:16.951386 22143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:16.942393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54535 (local address 127.0.98.62:43669)
0927 15:01:16.942888 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:16.942908 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:16.942950 (+ 42us) server_negotiation.cc:407] Connection header received
0927 15:01:16.943828 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:16.943852 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:16.944333 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:16.944707 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:16.945156 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:16.946464 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:16.947457 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:16.948139 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:16.948299 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:16.949199 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:16.949206 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:16.949226 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:01:16.949739 (+ 513us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:16.949810 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:01:16.949840 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:16.949978 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:16.950187 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 15:01:16.950345 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":136,"threads_started":1}
I20260927 15:01:16.951761 22140 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdd6563ef09445a381cb7a01837863fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd6563ef09445a381cb7a01837863fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43669 } }
I20260927 15:01:16.951985 22140 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdd6563ef09445a381cb7a01837863fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:01:16.952020 22140 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdd6563ef09445a381cb7a01837863fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd6563ef09445a381cb7a01837863fc, State: Initialized, Role: FOLLOWER
I20260927 15:01:16.952461 22140 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdd6563ef09445a381cb7a01837863fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd6563ef09445a381cb7a01837863fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43669 } }
I20260927 15:01:16.952603 22140 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdd6563ef09445a381cb7a01837863fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:01:16.952666 22140 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdd6563ef09445a381cb7a01837863fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:01:16.952759 22140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdd6563ef09445a381cb7a01837863fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:01:16.953825 22140 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdd6563ef09445a381cb7a01837863fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd6563ef09445a381cb7a01837863fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43669 } }
I20260927 15:01:16.954160 22140 leader_election.cc:302] T 00000000000000000000000000000000 P cdd6563ef09445a381cb7a01837863fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdd6563ef09445a381cb7a01837863fc; no voters:
I20260927 15:01:16.954363 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:01:16.954528 22140 leader_election.cc:288] T 00000000000000000000000000000000 P cdd6563ef09445a381cb7a01837863fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:01:16.954602 22145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdd6563ef09445a381cb7a01837863fc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:01:16.954969 22145 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdd6563ef09445a381cb7a01837863fc [term 1 LEADER]: Becoming Leader. State: Replica: cdd6563ef09445a381cb7a01837863fc, State: Running, Role: LEADER
I20260927 15:01:16.955314 22145 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdd6563ef09445a381cb7a01837863fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd6563ef09445a381cb7a01837863fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43669 } }
I20260927 15:01:16.956347 22140 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdd6563ef09445a381cb7a01837863fc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:01:16.957152 22147 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd6563ef09445a381cb7a01837863fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdd6563ef09445a381cb7a01837863fc. Latest consensus state: current_term: 1 leader_uuid: "cdd6563ef09445a381cb7a01837863fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd6563ef09445a381cb7a01837863fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43669 } } }
I20260927 15:01:16.957304 22147 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd6563ef09445a381cb7a01837863fc [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:16.957588 22146 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd6563ef09445a381cb7a01837863fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdd6563ef09445a381cb7a01837863fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd6563ef09445a381cb7a01837863fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43669 } } }
I20260927 15:01:16.957660 22146 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd6563ef09445a381cb7a01837863fc [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:16.957667 22158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:01:16.961133 22158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:01:16.962778 22143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:16.956639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55531 (local address 127.0.98.62:43669)
0927 15:01:16.956756 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:16.956761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:16.958096 (+ 1335us) server_negotiation.cc:407] Connection header received
0927 15:01:16.958306 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:16.958310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:16.958360 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:16.958411 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:16.958845 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:16.959610 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:16.960732 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:16.961071 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:16.961175 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:16.962134 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:16.962138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:16.962141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:01:16.962154 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:16.962194 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:01:16.962207 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:16.962283 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:16.962602 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 15:01:16.962641 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:01:16.967279 22158 catalog_manager.cc:1347] Generated new cluster ID: 865099321cba44649c185652240f301c
I20260927 15:01:16.967367 22158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:01:16.985294 22158 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:01:16.986285 22158 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:01:16.992369 22158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdd6563ef09445a381cb7a01837863fc: Generated new TSK 0
I20260927 15:01:16.993333 22158 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:01:17.003595 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 22072
W20260927 15:01:17.008509 22057 connection.cc:535] client connection to 127.0.98.62:43669 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:43669): Connection reset by peer (error 104)
2026-09-27T15:01:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:01:17 dist-test-slave-pz0h krb5kdc[22171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:01:17 dist-test-slave-pz0h krb5kdc[22171](info): set up 2 sockets
Sep 27 15:01:17 dist-test-slave-pz0h krb5kdc[22171](info): commencing operation
krb5kdc: starting...
W20260927 15:01:17.974896 392 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.005s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 15:01:18 dist-test-slave-pz0h krb5kdc[22171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:01:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:01:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:01:18.097612 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43899
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:01:18.212423 22184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:01:18.212739 22184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:01:18.212785 22184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:01:18.216971 22184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:01:18.217062 22184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:01:18.217082 22184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:01:18.217098 22184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:01:18.217113 22184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:01:18.222993 22184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43899
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:01:18.224191 22184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:01:18.225219 22184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:01:18.234704 22192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:18.234704 22189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:01:18.234963 22184 server_base.cc:1034] running on GCE node
W20260927 15:01:18.234714 22190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:01:18.235759 22184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:01:18.237089 22184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:01:18.238307 22184 hybrid_clock.cc:648] HybridClock initialized: now 1790521278238281 us; error 49 us; skew 500 ppm
Sep 27 15:01:18 dist-test-slave-pz0h krb5kdc[22171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:01:18.242012 22184 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:01:18.243134 22184 webserver.cc:460] Webserver started at http://127.0.98.62:45721/ using document root <none> and password file <none>
I20260927 15:01:18.243785 22184 fs_manager.cc:359] Metadata directory not provided
I20260927 15:01:18.243856 22184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:01:18.244069 22184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:01:18.246306 22184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "3b146de2ce3d49c7b2f265fb08c7c95f"
format_stamp: "Formatted at 2026-09-27 15:01:18 on dist-test-slave-pz0h"
server_key: "0e83327ce968801bf6da8ea011079a43"
server_key_iv: "f58df1fa1670a0c7b1ca108c45d4476f"
server_key_version: "encryptionkey@0"
I20260927 15:01:18.246946 22184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "3b146de2ce3d49c7b2f265fb08c7c95f"
format_stamp: "Formatted at 2026-09-27 15:01:18 on dist-test-slave-pz0h"
server_key: "0e83327ce968801bf6da8ea011079a43"
server_key_iv: "f58df1fa1670a0c7b1ca108c45d4476f"
server_key_version: "encryptionkey@0"
I20260927 15:01:18.251075 22184 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:01:18.253616 22199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:01:18.255264 22184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:01:18.255402 22184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "3b146de2ce3d49c7b2f265fb08c7c95f"
format_stamp: "Formatted at 2026-09-27 15:01:18 on dist-test-slave-pz0h"
server_key: "0e83327ce968801bf6da8ea011079a43"
server_key_iv: "f58df1fa1670a0c7b1ca108c45d4476f"
server_key_version: "encryptionkey@0"
I20260927 15:01:18.255539 22184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:01:18.260417 22184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:01:18.263790 22184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:01:18.263985 22184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:01:18.271476 22184 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43899
I20260927 15:01:18.271531 22251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43899 every 8 connection(s)
I20260927 15:01:18.272547 22184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:01:18.274191 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 22184
I20260927 15:01:18.274340 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 15:01:18.274626 392 external_mini_cluster.cc:1428] Setting key 24a91856c342aa31dcf0a48a3b2db069
I20260927 15:01:18.275880 22252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:01:18.280500 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:18.276117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36757 (local address 127.0.98.62:43899)
0927 15:01:18.276521 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:18.276537 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:18.276582 (+ 45us) server_negotiation.cc:407] Connection header received
0927 15:01:18.277246 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:18.277264 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:18.277566 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:18.277865 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:18.278749 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:18.278754 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:18.278766 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:01:18.278778 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:18.278895 (+ 117us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:01:18.278919 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:18.279038 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:18.279150 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 15:01:18.279303 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":106,"threads_started":1}
I20260927 15:01:18.282524 22252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b146de2ce3d49c7b2f265fb08c7c95f: Bootstrap starting.
I20260927 15:01:18.283481 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:01:18.285292 22252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b146de2ce3d49c7b2f265fb08c7c95f: Neither blocks nor log segments found. Creating new log.
I20260927 15:01:18.286191 22252 log.cc:824] T 00000000000000000000000000000000 P 3b146de2ce3d49c7b2f265fb08c7c95f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:01:18.287405 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:18.285764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52191 (local address 127.0.98.62:43899)
0927 15:01:18.285893 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:18.285897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:18.286050 (+ 153us) server_negotiation.cc:407] Connection header received
0927 15:01:18.286152 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:18.286155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:18.286197 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:18.286290 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:18.287017 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:18.287020 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:18.287023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:01:18.287036 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:18.287064 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:01:18.287077 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:18.287169 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:18.287248 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 15:01:18.287289 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:01:18.288587 22252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b146de2ce3d49c7b2f265fb08c7c95f: No bootstrap required, opened a new log
I20260927 15:01:18.292308 22252 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b146de2ce3d49c7b2f265fb08c7c95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b146de2ce3d49c7b2f265fb08c7c95f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43899 } }
I20260927 15:01:18.292588 22252 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b146de2ce3d49c7b2f265fb08c7c95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:01:18.292645 22252 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b146de2ce3d49c7b2f265fb08c7c95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b146de2ce3d49c7b2f265fb08c7c95f, State: Initialized, Role: FOLLOWER
I20260927 15:01:18.293047 22252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b146de2ce3d49c7b2f265fb08c7c95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b146de2ce3d49c7b2f265fb08c7c95f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43899 } }
I20260927 15:01:18.293182 22252 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b146de2ce3d49c7b2f265fb08c7c95f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:01:18.293251 22252 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b146de2ce3d49c7b2f265fb08c7c95f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:01:18.293352 22252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b146de2ce3d49c7b2f265fb08c7c95f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:01:18.294384 22252 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b146de2ce3d49c7b2f265fb08c7c95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b146de2ce3d49c7b2f265fb08c7c95f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43899 } }
I20260927 15:01:18.294754 22252 leader_election.cc:302] T 00000000000000000000000000000000 P 3b146de2ce3d49c7b2f265fb08c7c95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b146de2ce3d49c7b2f265fb08c7c95f; no voters:
I20260927 15:01:18.295145 22252 leader_election.cc:288] T 00000000000000000000000000000000 P 3b146de2ce3d49c7b2f265fb08c7c95f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:01:18.295277 22262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b146de2ce3d49c7b2f265fb08c7c95f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:01:18.295477 22262 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b146de2ce3d49c7b2f265fb08c7c95f [term 1 LEADER]: Becoming Leader. State: Replica: 3b146de2ce3d49c7b2f265fb08c7c95f, State: Running, Role: LEADER
I20260927 15:01:18.295823 22262 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b146de2ce3d49c7b2f265fb08c7c95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b146de2ce3d49c7b2f265fb08c7c95f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43899 } }
I20260927 15:01:18.296367 22252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b146de2ce3d49c7b2f265fb08c7c95f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:01:18.297605 22263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b146de2ce3d49c7b2f265fb08c7c95f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b146de2ce3d49c7b2f265fb08c7c95f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b146de2ce3d49c7b2f265fb08c7c95f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43899 } } }
I20260927 15:01:18.297658 22264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b146de2ce3d49c7b2f265fb08c7c95f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b146de2ce3d49c7b2f265fb08c7c95f. Latest consensus state: current_term: 1 leader_uuid: "3b146de2ce3d49c7b2f265fb08c7c95f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b146de2ce3d49c7b2f265fb08c7c95f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43899 } } }
I20260927 15:01:18.297732 22263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b146de2ce3d49c7b2f265fb08c7c95f [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:18.297756 22264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b146de2ce3d49c7b2f265fb08c7c95f [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:18.298158 22271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:01:18.301596 22271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:01:18.307273 22271 catalog_manager.cc:1347] Generated new cluster ID: 646dc9aad1b54891bf22d7a07f9bb134
I20260927 15:01:18.307360 22271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:01:18.328311 22271 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:01:18.328739 22271 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:01:18.345638 22271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b146de2ce3d49c7b2f265fb08c7c95f: Generated new TSK 0
I20260927 15:01:18.346642 22271 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:01:18.361495 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 22184
2026-09-27T15:01:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:01:18 dist-test-slave-pz0h krb5kdc[22283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:01:18 dist-test-slave-pz0h krb5kdc[22283](info): set up 2 sockets
Sep 27 15:01:18 dist-test-slave-pz0h krb5kdc[22283](info): commencing operation
krb5kdc: starting...
W20260927 15:01:19.326962 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 15:01:19 dist-test-slave-pz0h krb5kdc[22283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:01:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:01:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:01:19.449863 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42427
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:01:19.565896 22296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:01:19.566210 22296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:01:19.566272 22296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:01:19.570509 22296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:01:19.570606 22296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:01:19.570634 22296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:01:19.570655 22296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:01:19.570674 22296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:01:19.576524 22296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:01:19.578161 22296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:01:19.579313 22296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:01:19.588665 22302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:19.588676 22301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:19.588665 22304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:01:19.588918 22296 server_base.cc:1034] running on GCE node
I20260927 15:01:19.589633 22296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:01:19.590919 22296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:01:19.592109 22296 hybrid_clock.cc:648] HybridClock initialized: now 1790521279592094 us; error 37 us; skew 500 ppm
Sep 27 15:01:19 dist-test-slave-pz0h krb5kdc[22283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:01:19.601640 22296 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:01:19.602746 22296 webserver.cc:460] Webserver started at http://127.0.98.62:46607/ using document root <none> and password file <none>
I20260927 15:01:19.603367 22296 fs_manager.cc:359] Metadata directory not provided
I20260927 15:01:19.603456 22296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:01:19.603669 22296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:01:19.605906 22296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "a0af69df7cf74a1180c7be1db43b1ad7"
format_stamp: "Formatted at 2026-09-27 15:01:19 on dist-test-slave-pz0h"
server_key: "16cbc612668d274aa58eb2ad1413cdb1"
server_key_iv: "cd35dbe6e2db4aeb090a3eb7028d1101"
server_key_version: "encryptionkey@0"
I20260927 15:01:19.606528 22296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "a0af69df7cf74a1180c7be1db43b1ad7"
format_stamp: "Formatted at 2026-09-27 15:01:19 on dist-test-slave-pz0h"
server_key: "16cbc612668d274aa58eb2ad1413cdb1"
server_key_iv: "cd35dbe6e2db4aeb090a3eb7028d1101"
server_key_version: "encryptionkey@0"
I20260927 15:01:19.610553 22296 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260927 15:01:19.613399 22311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:01:19.614773 22296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:01:19.614974 22296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "a0af69df7cf74a1180c7be1db43b1ad7"
format_stamp: "Formatted at 2026-09-27 15:01:19 on dist-test-slave-pz0h"
server_key: "16cbc612668d274aa58eb2ad1413cdb1"
server_key_iv: "cd35dbe6e2db4aeb090a3eb7028d1101"
server_key_version: "encryptionkey@0"
I20260927 15:01:19.615100 22296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:01:19.634699 22296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:01:19.638031 22296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:01:19.638239 22296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:01:19.645967 22296 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42427
I20260927 15:01:19.645989 22363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42427 every 8 connection(s)
I20260927 15:01:19.646988 22296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:01:19.649914 22364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:01:19.656670 22364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0af69df7cf74a1180c7be1db43b1ad7: Bootstrap starting.
I20260927 15:01:19.657269 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 22296
I20260927 15:01:19.657406 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 15:01:19.657653 392 external_mini_cluster.cc:1428] Setting key 3ce1ec384ca70d608fa498873e39e79b
I20260927 15:01:19.660223 22364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0af69df7cf74a1180c7be1db43b1ad7: Neither blocks nor log segments found. Creating new log.
I20260927 15:01:19.661010 22364 log.cc:824] T 00000000000000000000000000000000 P a0af69df7cf74a1180c7be1db43b1ad7: Log is configured to *not* fsync() on all Append() calls
I20260927 15:01:19.663164 22364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0af69df7cf74a1180c7be1db43b1ad7: No bootstrap required, opened a new log
I20260927 15:01:19.667222 22364 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0af69df7cf74a1180c7be1db43b1ad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0af69df7cf74a1180c7be1db43b1ad7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42427 } }
Sep 27 15:01:19 dist-test-slave-pz0h krb5kdc[22283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:01:19.667583 22364 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0af69df7cf74a1180c7be1db43b1ad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:01:19.667647 22364 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0af69df7cf74a1180c7be1db43b1ad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0af69df7cf74a1180c7be1db43b1ad7, State: Initialized, Role: FOLLOWER
I20260927 15:01:19.668184 22364 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0af69df7cf74a1180c7be1db43b1ad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0af69df7cf74a1180c7be1db43b1ad7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42427 } }
I20260927 15:01:19.668334 22364 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0af69df7cf74a1180c7be1db43b1ad7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:01:19.668406 22364 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0af69df7cf74a1180c7be1db43b1ad7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:01:19.668509 22364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0af69df7cf74a1180c7be1db43b1ad7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:01:19.669687 22364 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0af69df7cf74a1180c7be1db43b1ad7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0af69df7cf74a1180c7be1db43b1ad7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42427 } }
I20260927 15:01:19.670068 22364 leader_election.cc:302] T 00000000000000000000000000000000 P a0af69df7cf74a1180c7be1db43b1ad7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0af69df7cf74a1180c7be1db43b1ad7; no voters:
I20260927 15:01:19.670426 22364 leader_election.cc:288] T 00000000000000000000000000000000 P a0af69df7cf74a1180c7be1db43b1ad7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:01:19.670553 22369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0af69df7cf74a1180c7be1db43b1ad7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:01:19.670864 22369 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0af69df7cf74a1180c7be1db43b1ad7 [term 1 LEADER]: Becoming Leader. State: Replica: a0af69df7cf74a1180c7be1db43b1ad7, State: Running, Role: LEADER
I20260927 15:01:19.671226 22369 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0af69df7cf74a1180c7be1db43b1ad7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0af69df7cf74a1180c7be1db43b1ad7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42427 } }
I20260927 15:01:19.671905 22364 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0af69df7cf74a1180c7be1db43b1ad7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:01:19.672501 22369 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0af69df7cf74a1180c7be1db43b1ad7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0af69df7cf74a1180c7be1db43b1ad7. Latest consensus state: current_term: 1 leader_uuid: "a0af69df7cf74a1180c7be1db43b1ad7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0af69df7cf74a1180c7be1db43b1ad7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42427 } } }
I20260927 15:01:19.672670 22369 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0af69df7cf74a1180c7be1db43b1ad7 [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:19.672948 22370 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0af69df7cf74a1180c7be1db43b1ad7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0af69df7cf74a1180c7be1db43b1ad7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0af69df7cf74a1180c7be1db43b1ad7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42427 } } }
I20260927 15:01:19.673110 22370 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0af69df7cf74a1180c7be1db43b1ad7 [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:19.675052 22367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:19.659418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55525 (local address 127.0.98.62:42427)
0927 15:01:19.659912 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:19.659928 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:19.659966 (+ 38us) server_negotiation.cc:407] Connection header received
0927 15:01:19.660727 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:19.660748 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:19.661155 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:19.661486 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:19.662330 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:19.663438 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:19.664566 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:19.665094 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:19.665268 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:19.668314 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:19.668320 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:19.668337 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:19.668381 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:19.670526 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:19.671154 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:19.671161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:19.671168 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:19.671337 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:19.671691 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:19.671695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:19.671697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:19.672268 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:19.672489 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:19.673469 (+ 980us) server_negotiation.cc:299] Negotiation successful
0927 15:01:19.673870 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":163,"threads_started":1}
W20260927 15:01:19.677192 22384 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a0af69df7cf74a1180c7be1db43b1ad7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:01:19.677277 22384 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:01:19.677420 22376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:01:19.678371 22376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:01:19.678838 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:01:19.685292 22376 catalog_manager.cc:1347] Generated new cluster ID: 98801d3fd1bb425c996b84c99ddb5d52
I20260927 15:01:19.685467 22376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:01:19.692709 22367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:19.680786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50639 (local address 127.0.98.62:42427)
0927 15:01:19.680910 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:19.680915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:19.683238 (+ 2323us) server_negotiation.cc:407] Connection header received
0927 15:01:19.683327 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:19.683330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:19.683381 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:19.683448 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:19.684289 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:19.685173 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:19.686437 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:19.686853 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:19.686952 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:19.688608 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:19.688612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:19.688615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:19.688643 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:19.690262 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:19.690833 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:19.690836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:19.690838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:19.691015 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:19.691518 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:19.691521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:19.691523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:19.691769 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:19.691919 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:19.692193 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 15:01:19.692367 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 15:01:19.707180 22376 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:01:19.708629 22376 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:01:19.739089 22376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0af69df7cf74a1180c7be1db43b1ad7: Generated new TSK 0
I20260927 15:01:19.739916 22376 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:01:19.751288 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 22296
2026-09-27T15:01:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1389 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:01:19 dist-test-slave-pz0h krb5kdc[22395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:01:19 dist-test-slave-pz0h krb5kdc[22395](info): set up 2 sockets
Sep 27 15:01:19 dist-test-slave-pz0h krb5kdc[22395](info): commencing operation
krb5kdc: starting...
W20260927 15:01:20.718933 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 15:01:20 dist-test-slave-pz0h krb5kdc[22395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:01:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:01:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:01:20.839860 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39705
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:01:20.958204 22408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:01:20.958528 22408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:01:20.958588 22408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:01:20.962754 22408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:01:20.962867 22408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:01:20.962895 22408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:01:20.962915 22408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:01:20.962935 22408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:01:20.968534 22408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:01:20.969707 22408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:01:20.970752 22408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:01:20.979223 22416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:20.979251 22414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:01:20.979383 22408 server_base.cc:1034] running on GCE node
W20260927 15:01:20.979223 22413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:01:20.980145 22408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:01:20.981303 22408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:01:20.982491 22408 hybrid_clock.cc:648] HybridClock initialized: now 1790521280982478 us; error 28 us; skew 500 ppm
Sep 27 15:01:20 dist-test-slave-pz0h krb5kdc[22395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:01:20.985374 22408 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:01:20.986377 22408 webserver.cc:460] Webserver started at http://127.0.98.62:39685/ using document root <none> and password file <none>
I20260927 15:01:20.986976 22408 fs_manager.cc:359] Metadata directory not provided
I20260927 15:01:20.987046 22408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:01:20.987241 22408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:01:20.989347 22408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "d9b5287765854b16bbca60f1094168fa"
format_stamp: "Formatted at 2026-09-27 15:01:20 on dist-test-slave-pz0h"
server_key: "59300fbc9142c4a22ad8156cbd798880"
server_key_iv: "f8920f37deacf175b4d344669074fe91"
server_key_version: "encryptionkey@0"
I20260927 15:01:20.989909 22408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "d9b5287765854b16bbca60f1094168fa"
format_stamp: "Formatted at 2026-09-27 15:01:20 on dist-test-slave-pz0h"
server_key: "59300fbc9142c4a22ad8156cbd798880"
server_key_iv: "f8920f37deacf175b4d344669074fe91"
server_key_version: "encryptionkey@0"
I20260927 15:01:20.993978 22408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260927 15:01:20.996933 22423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:01:20.998279 22408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 15:01:20.998406 22408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "d9b5287765854b16bbca60f1094168fa"
format_stamp: "Formatted at 2026-09-27 15:01:20 on dist-test-slave-pz0h"
server_key: "59300fbc9142c4a22ad8156cbd798880"
server_key_iv: "f8920f37deacf175b4d344669074fe91"
server_key_version: "encryptionkey@0"
I20260927 15:01:20.998538 22408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:01:21.024302 22408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:01:21.027976 22408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:01:21.028162 22408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:01:21.035414 22408 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39705
I20260927 15:01:21.035424 22475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39705 every 8 connection(s)
I20260927 15:01:21.036414 22408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:01:21.036743 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 22408
I20260927 15:01:21.036883 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 15:01:21.037184 392 external_mini_cluster.cc:1428] Setting key 731a2596bb68ee8800f23f469753a2aa
I20260927 15:01:21.039655 22476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:01:21.045506 22476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9b5287765854b16bbca60f1094168fa: Bootstrap starting.
I20260927 15:01:21.046072 22479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:21.038609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34875 (local address 127.0.98.62:39705)
0927 15:01:21.039042 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:21.039056 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:21.039090 (+ 34us) server_negotiation.cc:407] Connection header received
0927 15:01:21.039737 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:21.039761 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:21.040127 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:21.040440 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:21.040828 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:21.041884 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:21.042787 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:21.043304 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:21.043472 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:21.044302 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:21.044307 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:21.044320 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:01:21.044736 (+ 416us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:21.044799 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:01:21.044822 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:21.044910 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:21.045054 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 15:01:21.045268 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":157,"threads_started":1}
I20260927 15:01:21.048038 22476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9b5287765854b16bbca60f1094168fa: Neither blocks nor log segments found. Creating new log.
I20260927 15:01:21.048894 22476 log.cc:824] T 00000000000000000000000000000000 P d9b5287765854b16bbca60f1094168fa: Log is configured to *not* fsync() on all Append() calls
I20260927 15:01:21.049274 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:01:21.051015 22476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9b5287765854b16bbca60f1094168fa: No bootstrap required, opened a new log
I20260927 15:01:21.055418 22479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:21.051211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38775 (local address 127.0.98.62:39705)
0927 15:01:21.051342 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:21.051346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:21.051554 (+ 208us) server_negotiation.cc:407] Connection header received
0927 15:01:21.051645 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:21.051648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:21.051699 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:21.051801 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:21.052007 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:21.052768 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:21.053663 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:21.054015 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:21.054122 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:21.054983 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:21.054987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:21.054989 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:01:21.055004 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:21.055037 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:01:21.055051 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:21.055120 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:21.055202 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 15:01:21.055266 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:01:21.055557 22476 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9b5287765854b16bbca60f1094168fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b5287765854b16bbca60f1094168fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39705 } }
I20260927 15:01:21.055840 22476 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9b5287765854b16bbca60f1094168fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:01:21.055909 22476 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9b5287765854b16bbca60f1094168fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9b5287765854b16bbca60f1094168fa, State: Initialized, Role: FOLLOWER
I20260927 15:01:21.056408 22476 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9b5287765854b16bbca60f1094168fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b5287765854b16bbca60f1094168fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39705 } }
I20260927 15:01:21.056568 22476 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9b5287765854b16bbca60f1094168fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:01:21.056644 22476 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9b5287765854b16bbca60f1094168fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:01:21.056751 22476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9b5287765854b16bbca60f1094168fa [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:01:21.057739 22476 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9b5287765854b16bbca60f1094168fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b5287765854b16bbca60f1094168fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39705 } }
I20260927 15:01:21.058106 22476 leader_election.cc:302] T 00000000000000000000000000000000 P d9b5287765854b16bbca60f1094168fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9b5287765854b16bbca60f1094168fa; no voters:
I20260927 15:01:21.058432 22476 leader_election.cc:288] T 00000000000000000000000000000000 P d9b5287765854b16bbca60f1094168fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:01:21.058576 22486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9b5287765854b16bbca60f1094168fa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:01:21.058835 22486 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9b5287765854b16bbca60f1094168fa [term 1 LEADER]: Becoming Leader. State: Replica: d9b5287765854b16bbca60f1094168fa, State: Running, Role: LEADER
I20260927 15:01:21.059155 22486 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9b5287765854b16bbca60f1094168fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b5287765854b16bbca60f1094168fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39705 } }
I20260927 15:01:21.059688 22476 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9b5287765854b16bbca60f1094168fa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:01:21.060869 22488 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9b5287765854b16bbca60f1094168fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9b5287765854b16bbca60f1094168fa. Latest consensus state: current_term: 1 leader_uuid: "d9b5287765854b16bbca60f1094168fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b5287765854b16bbca60f1094168fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39705 } } }
I20260927 15:01:21.061045 22488 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9b5287765854b16bbca60f1094168fa [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:21.060881 22487 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9b5287765854b16bbca60f1094168fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9b5287765854b16bbca60f1094168fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b5287765854b16bbca60f1094168fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39705 } } }
I20260927 15:01:21.061142 22487 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9b5287765854b16bbca60f1094168fa [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:21.061473 22496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:01:21.065212 22496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:01:21.071009 22496 catalog_manager.cc:1347] Generated new cluster ID: 73e20e05b30f40d79bcaeb3ad73bdb5b
I20260927 15:01:21.071103 22496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:01:21.085450 22496 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:01:21.086784 22496 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:01:21.097011 22496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9b5287765854b16bbca60f1094168fa: Generated new TSK 0
I20260927 15:01:21.097952 22496 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:01:21.105397 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 22408
2026-09-27T15:01:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1354 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:01:21 dist-test-slave-pz0h krb5kdc[22507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:01:21 dist-test-slave-pz0h krb5kdc[22507](info): set up 2 sockets
Sep 27 15:01:21 dist-test-slave-pz0h krb5kdc[22507](info): commencing operation
krb5kdc: starting...
W20260927 15:01:22.067602 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 15:01:22 dist-test-slave-pz0h krb5kdc[22507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:01:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:01:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:01:22.194674 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36703
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:01:22.309032 22520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:01:22.309350 22520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:01:22.309409 22520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:01:22.313660 22520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:01:22.313750 22520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:01:22.313777 22520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:01:22.313799 22520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:01:22.313818 22520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:01:22.319605 22520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:01:22.320812 22520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:01:22.321866 22520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:01:22.331176 22526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:22.331350 22525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:22.331175 22528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:01:22.331506 22520 server_base.cc:1034] running on GCE node
I20260927 15:01:22.332181 22520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:01:22.333357 22520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:01:22.334547 22520 hybrid_clock.cc:648] HybridClock initialized: now 1790521282334522 us; error 45 us; skew 500 ppm
Sep 27 15:01:22 dist-test-slave-pz0h krb5kdc[22507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:01:22.337462 22520 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:01:22.338495 22520 webserver.cc:460] Webserver started at http://127.0.98.62:39367/ using document root <none> and password file <none>
I20260927 15:01:22.339136 22520 fs_manager.cc:359] Metadata directory not provided
I20260927 15:01:22.339214 22520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:01:22.339416 22520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:01:22.341677 22520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "766ccd7fe204493084cfbdf5d5500456"
format_stamp: "Formatted at 2026-09-27 15:01:22 on dist-test-slave-pz0h"
server_key: "b873a923c28da85f28d09475fb434d76"
server_key_iv: "0179db59b987ef3233f3e15666addfa1"
server_key_version: "encryptionkey@0"
I20260927 15:01:22.342311 22520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "766ccd7fe204493084cfbdf5d5500456"
format_stamp: "Formatted at 2026-09-27 15:01:22 on dist-test-slave-pz0h"
server_key: "b873a923c28da85f28d09475fb434d76"
server_key_iv: "0179db59b987ef3233f3e15666addfa1"
server_key_version: "encryptionkey@0"
I20260927 15:01:22.347258 22520 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260927 15:01:22.350201 22535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:01:22.351737 22520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:01:22.351878 22520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "766ccd7fe204493084cfbdf5d5500456"
format_stamp: "Formatted at 2026-09-27 15:01:22 on dist-test-slave-pz0h"
server_key: "b873a923c28da85f28d09475fb434d76"
server_key_iv: "0179db59b987ef3233f3e15666addfa1"
server_key_version: "encryptionkey@0"
I20260927 15:01:22.351999 22520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:01:22.356940 22520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:01:22.360216 22520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:01:22.360404 22520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:01:22.367672 22520 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36703
I20260927 15:01:22.367715 22587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36703 every 8 connection(s)
I20260927 15:01:22.368692 22520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:01:22.371430 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 22520
I20260927 15:01:22.371583 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 15:01:22.371882 392 external_mini_cluster.cc:1428] Setting key 92598309e8a7827502fabe5fd169675c
I20260927 15:01:22.372104 22588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:01:22.378008 22591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:22.373382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58329 (local address 127.0.98.62:36703)
0927 15:01:22.373897 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:22.373916 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:22.373960 (+ 44us) server_negotiation.cc:407] Connection header received
0927 15:01:22.374993 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:22.375019 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:22.375436 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:22.375832 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:22.376524 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:22.376530 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:22.376544 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:01:22.376555 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:22.376612 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:01:22.376635 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:22.376802 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:22.376869 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 15:01:22.377030 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":182,"threads_started":1}
I20260927 15:01:22.380153 22588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 766ccd7fe204493084cfbdf5d5500456: Bootstrap starting.
I20260927 15:01:22.381269 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:01:22.382861 22588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 766ccd7fe204493084cfbdf5d5500456: Neither blocks nor log segments found. Creating new log.
I20260927 15:01:22.384332 22588 log.cc:824] T 00000000000000000000000000000000 P 766ccd7fe204493084cfbdf5d5500456: Log is configured to *not* fsync() on all Append() calls
I20260927 15:01:22.385043 22591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:22.383426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59043 (local address 127.0.98.62:36703)
0927 15:01:22.383598 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:22.383604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:22.383726 (+ 122us) server_negotiation.cc:407] Connection header received
0927 15:01:22.383791 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:22.383794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:22.383860 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:22.383963 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:22.384623 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:22.384627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:22.384630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:01:22.384643 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:22.384675 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:01:22.384689 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:22.384790 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:22.384874 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 15:01:22.384916 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:01:22.386754 22588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 766ccd7fe204493084cfbdf5d5500456: No bootstrap required, opened a new log
I20260927 15:01:22.390676 22588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 766ccd7fe204493084cfbdf5d5500456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "766ccd7fe204493084cfbdf5d5500456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36703 } }
I20260927 15:01:22.391032 22588 raft_consensus.cc:374] T 00000000000000000000000000000000 P 766ccd7fe204493084cfbdf5d5500456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:01:22.391096 22588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 766ccd7fe204493084cfbdf5d5500456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 766ccd7fe204493084cfbdf5d5500456, State: Initialized, Role: FOLLOWER
I20260927 15:01:22.391559 22588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 766ccd7fe204493084cfbdf5d5500456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "766ccd7fe204493084cfbdf5d5500456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36703 } }
I20260927 15:01:22.391700 22588 raft_consensus.cc:388] T 00000000000000000000000000000000 P 766ccd7fe204493084cfbdf5d5500456 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:01:22.391768 22588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 766ccd7fe204493084cfbdf5d5500456 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:01:22.391868 22588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 766ccd7fe204493084cfbdf5d5500456 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:01:22.392885 22588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 766ccd7fe204493084cfbdf5d5500456 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "766ccd7fe204493084cfbdf5d5500456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36703 } }
I20260927 15:01:22.393252 22588 leader_election.cc:302] T 00000000000000000000000000000000 P 766ccd7fe204493084cfbdf5d5500456 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 766ccd7fe204493084cfbdf5d5500456; no voters:
I20260927 15:01:22.393597 22588 leader_election.cc:288] T 00000000000000000000000000000000 P 766ccd7fe204493084cfbdf5d5500456 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:01:22.393863 22598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 766ccd7fe204493084cfbdf5d5500456 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:01:22.394591 22598 raft_consensus.cc:686] T 00000000000000000000000000000000 P 766ccd7fe204493084cfbdf5d5500456 [term 1 LEADER]: Becoming Leader. State: Replica: 766ccd7fe204493084cfbdf5d5500456, State: Running, Role: LEADER
I20260927 15:01:22.395092 22588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 766ccd7fe204493084cfbdf5d5500456 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:01:22.395108 22598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 766ccd7fe204493084cfbdf5d5500456 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "766ccd7fe204493084cfbdf5d5500456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36703 } }
I20260927 15:01:22.396953 22600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 766ccd7fe204493084cfbdf5d5500456 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 766ccd7fe204493084cfbdf5d5500456. Latest consensus state: current_term: 1 leader_uuid: "766ccd7fe204493084cfbdf5d5500456" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "766ccd7fe204493084cfbdf5d5500456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36703 } } }
I20260927 15:01:22.397111 22600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 766ccd7fe204493084cfbdf5d5500456 [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:22.397419 22599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 766ccd7fe204493084cfbdf5d5500456 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "766ccd7fe204493084cfbdf5d5500456" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "766ccd7fe204493084cfbdf5d5500456" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36703 } } }
I20260927 15:01:22.397531 22599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 766ccd7fe204493084cfbdf5d5500456 [sys.catalog]: This master's current role is: LEADER
W20260927 15:01:22.399971 22612 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 766ccd7fe204493084cfbdf5d5500456: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:01:22.400045 22612 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:01:22.400171 22613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:01:22.401117 22613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:01:22.407024 22613 catalog_manager.cc:1347] Generated new cluster ID: ada39387d58240d2b2746393a9f8794e
I20260927 15:01:22.407116 22613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:01:22.428437 22613 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:01:22.428853 22613 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:01:22.443724 22613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 766ccd7fe204493084cfbdf5d5500456: Generated new TSK 0
I20260927 15:01:22.444528 22613 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:01:22.452965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 22520
2026-09-27T15:01:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1348 ms)
[----------] 18 tests from AuthTokenIssuingTest (24122 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:01:22 dist-test-slave-pz0h krb5kdc[22619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:01:22 dist-test-slave-pz0h krb5kdc[22619](info): set up 2 sockets
Sep 27 15:01:22 dist-test-slave-pz0h krb5kdc[22619](info): commencing operation
krb5kdc: starting...
W20260927 15:01:23.419343 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 15:01:23 dist-test-slave-pz0h krb5kdc[22619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:01:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:01:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:01:23.542559 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34643
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34643,127.0.98.61:37189,127.0.98.60:37239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:01:23.654693 22632 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:01:23.655076 22632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:01:23.655112 22632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:01:23.655184 22632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:01:23.659444 22632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:01:23.659525 22632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:01:23.659551 22632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:01:23.659572 22632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:01:23.659591 22632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:01:23.665261 22632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34643,127.0.98.61:37189,127.0.98.60:37239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:01:23.666425 22632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:01:23.667496 22632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:01:23.676764 22640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:23.676764 22637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:23.676928 22638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:01:23.677220 22632 server_base.cc:1034] running on GCE node
I20260927 15:01:23.677955 22632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:01:23.679267 22632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:01:23.680492 22632 hybrid_clock.cc:648] HybridClock initialized: now 1790521283680451 us; error 57 us; skew 500 ppm
Sep 27 15:01:23 dist-test-slave-pz0h krb5kdc[22619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:01:23.683813 22632 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:01:23.684893 22632 webserver.cc:460] Webserver started at http://127.0.98.62:42543/ using document root <none> and password file <none>
I20260927 15:01:23.685477 22632 fs_manager.cc:359] Metadata directory not provided
I20260927 15:01:23.685551 22632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:01:23.685760 22632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:01:23.688030 22632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "1524b2a471f3404c8429b410474a3375"
format_stamp: "Formatted at 2026-09-27 15:01:23 on dist-test-slave-pz0h"
server_key: "b61f46c1d196116a08659459dcdc3196"
server_key_iv: "a4dc4f7a61746bdf6b584440f70c19cc"
server_key_version: "encryptionkey@0"
I20260927 15:01:23.688633 22632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "1524b2a471f3404c8429b410474a3375"
format_stamp: "Formatted at 2026-09-27 15:01:23 on dist-test-slave-pz0h"
server_key: "b61f46c1d196116a08659459dcdc3196"
server_key_iv: "a4dc4f7a61746bdf6b584440f70c19cc"
server_key_version: "encryptionkey@0"
I20260927 15:01:23.692986 22632 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 15:01:23.695879 22647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:01:23.697252 22632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:01:23.697383 22632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "1524b2a471f3404c8429b410474a3375"
format_stamp: "Formatted at 2026-09-27 15:01:23 on dist-test-slave-pz0h"
server_key: "b61f46c1d196116a08659459dcdc3196"
server_key_iv: "a4dc4f7a61746bdf6b584440f70c19cc"
server_key_version: "encryptionkey@0"
I20260927 15:01:23.697530 22632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:01:23.722895 22632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:01:23.726359 22632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:01:23.726550 22632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:01:23.733933 22632 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34643
I20260927 15:01:23.734205 22699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34643 every 8 connection(s)
I20260927 15:01:23.734979 22632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:01:23.737953 22700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:01:23.739609 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 22632
I20260927 15:01:23.739735 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 15:01:23.740036 392 external_mini_cluster.cc:1428] Setting key 9c356cebfbbc3b40224fbe73f6f61bbc
I20260927 15:01:23.742743 22700 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34643 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 15:01:23 dist-test-slave-pz0h krb5kdc[22619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:01:23.758829 22704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:23.745295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59974 (local address 127.0.98.62:34643)
0927 15:01:23.745890 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:23.745904 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:23.746023 (+ 119us) server_negotiation.cc:407] Connection header received
0927 15:01:23.747241 (+ 1218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:23.747248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:23.747529 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:23.747756 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:23.748718 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:23.749648 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:23.750762 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:23.751209 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:23.751381 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:23.753885 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:23.753891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:23.753903 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:23.753928 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:23.755551 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:23.756031 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:23.756036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:23.756037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:23.756143 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:23.756429 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:23.756432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:23.756434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:23.756889 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:23.757014 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:23.757396 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 15:01:23.757679 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":470,"spinlock_wait_cycles":37632,"thread_start_us":166,"threads_started":1}
I20260927 15:01:23.758857 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:23.745295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34643 (local address 127.0.0.1:59974)
0927 15:01:23.745839 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:23.745890 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 15:01:23.746877 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:01:23.747800 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:01:23.747806 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:01:23.748184 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:01:23.748508 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:23.748525 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:23.749789 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:23.749792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:23.750598 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:23.750610 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:23.751378 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:23.751383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:23.751497 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:23.752143 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:01:23.752162 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:01:23.753715 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:01:23.755677 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:23.755686 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:23.755698 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:23.755924 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:23.756235 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:23.756238 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:23.756240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:23.756347 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:23.757014 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:01:23.757020 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:01:23.757154 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 15:01:23.757319 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 15:01:23.757742 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":102,"thread_start_us":127,"threads_started":1}
I20260927 15:01:23.761459 22700 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37189 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:01:23.762308 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:23.761988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37189 (local address 127.0.0.1:51734)
0927 15:01:23.762147 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:23.762232 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 15:01:23.762501 22650 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37189: connect: Connection refused (error 111)
I20260927 15:01:23.763504 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:23.763251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37189 (local address 127.0.0.1:51748)
0927 15:01:23.763363 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:23.763413 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 15:01:23.763626 22700 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37189: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37189: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 15:01:23.765365 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37189
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34643,127.0.98.61:37189,127.0.98.60:37239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:01:23.812956 22700 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37189 } attempt: 1
I20260927 15:01:23.814174 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:23.813773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37189 (local address 127.0.0.1:51764)
0927 15:01:23.813983 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:23.814059 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 15:01:23.815088 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:23.814860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37189 (local address 127.0.0.1:51772)
0927 15:01:23.814967 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:23.815017 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 15:01:23.815224 22700 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37189: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37189: connect: Connection refused (error 111)
I20260927 15:01:23.883397 22700 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37189 } attempt: 2
I20260927 15:01:23.884342 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:23.884016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37189 (local address 127.0.0.1:51780)
0927 15:01:23.884192 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:23.884261 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 15:01:23.885236 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:23.884960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37189 (local address 127.0.0.1:51788)
0927 15:01:23.885086 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:23.885147 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 15:01:23.885404 22700 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37189: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37189: connect: Connection refused (error 111)
W20260927 15:01:23.890054 22707 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:01:23.890400 22707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:01:23.890424 22707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:01:23.890463 22707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:01:23.894774 22707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:01:23.894924 22707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:01:23.894948 22707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:01:23.894963 22707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:01:23.894979 22707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:01:23.900810 22707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34643,127.0.98.61:37189,127.0.98.60:37239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:01:23.902230 22707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:01:23.903569 22707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:01:23.912873 22713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:23.912873 22712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:23.913130 22715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:01:23.913341 22707 server_base.cc:1034] running on GCE node
I20260927 15:01:23.913982 22707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:01:23.915277 22707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:01:23.916481 22707 hybrid_clock.cc:648] HybridClock initialized: now 1790521283916459 us; error 42 us; skew 500 ppm
Sep 27 15:01:23 dist-test-slave-pz0h krb5kdc[22619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:01:23.920336 22707 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 15:01:23.921579 22707 webserver.cc:460] Webserver started at http://127.0.98.61:41375/ using document root <none> and password file <none>
I20260927 15:01:23.922549 22707 fs_manager.cc:359] Metadata directory not provided
I20260927 15:01:23.922667 22707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:01:23.922951 22707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:01:23.925226 22707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-1/data/instance:
uuid: "075139e2a39e43109553a940df31de5b"
format_stamp: "Formatted at 2026-09-27 15:01:23 on dist-test-slave-pz0h"
server_key: "d9971d2795719a37f60a76b74735f5ad"
server_key_iv: "899559fe1b4d9f6ef7e036af3be655f7"
server_key_version: "encryptionkey@0"
I20260927 15:01:23.925832 22707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-1/wal/instance:
uuid: "075139e2a39e43109553a940df31de5b"
format_stamp: "Formatted at 2026-09-27 15:01:23 on dist-test-slave-pz0h"
server_key: "d9971d2795719a37f60a76b74735f5ad"
server_key_iv: "899559fe1b4d9f6ef7e036af3be655f7"
server_key_version: "encryptionkey@0"
I20260927 15:01:23.929842 22707 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 15:01:23.932572 22722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:01:23.933905 22707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:01:23.934043 22707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-1/wal
uuid: "075139e2a39e43109553a940df31de5b"
format_stamp: "Formatted at 2026-09-27 15:01:23 on dist-test-slave-pz0h"
server_key: "d9971d2795719a37f60a76b74735f5ad"
server_key_iv: "899559fe1b4d9f6ef7e036af3be655f7"
server_key_version: "encryptionkey@0"
I20260927 15:01:23.934161 22707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:01:23.954370 22707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:01:23.958505 22707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:01:23.958741 22707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:01:23.966555 22707 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37189
I20260927 15:01:23.966904 22774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37189 every 8 connection(s)
I20260927 15:01:23.967785 22707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-1/data/info.pb
I20260927 15:01:23.971812 22775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:01:23.972747 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 22707
I20260927 15:01:23.972862 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-1/wal/instance
I20260927 15:01:23.973104 392 external_mini_cluster.cc:1428] Setting key f3bd370dbf5bb01ddc205c9d6d1fdf87
I20260927 15:01:23.976605 22700 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37189 } attempt: 3
I20260927 15:01:23.976994 22775 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34643 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 15:01:23 dist-test-slave-pz0h krb5kdc[22619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 15:01:23 dist-test-slave-pz0h krb5kdc[22619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:01:23.992084 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:23.977236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37189 (local address 127.0.0.1:51802)
0927 15:01:23.977491 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:23.977508 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:01:23.977614 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:01:23.980659 (+ 3045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:01:23.980663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:01:23.980859 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:01:23.980987 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:23.980995 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:23.982939 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:23.982943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:23.983728 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:23.983741 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:23.984475 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:23.984478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:23.984575 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:23.985084 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:01:23.985099 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:01:23.986607 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:01:23.989146 (+ 2539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:23.989151 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:23.989153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:23.989509 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:23.989959 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:23.989963 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:23.989965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:23.990055 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:23.991375 (+ 1320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:01:23.991379 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:01:23.991561 (+ 182us) client_negotiation.cc:769] Sending connection context
0927 15:01:23.991703 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 15:01:23.991885 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260927 15:01:23.993029 22704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:23.979335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59978 (local address 127.0.98.62:34643)
0927 15:01:23.979465 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:23.979470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:23.979896 (+ 426us) server_negotiation.cc:407] Connection header received
0927 15:01:23.980377 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:23.980381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:23.980434 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:23.980506 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:23.981608 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:23.982154 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:23.983759 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:23.984118 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:23.984259 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:23.987572 (+ 3313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:23.987576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:23.987579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:23.987612 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:23.989640 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:23.990399 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:23.990405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:23.990407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:23.990592 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:23.991062 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:23.991066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:23.991067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:23.991323 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:23.991443 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:23.992638 (+ 1195us) server_negotiation.cc:299] Negotiation successful
0927 15:01:23.992844 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:01:23.993937 22778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:23.978231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51802 (local address 127.0.98.61:37189)
0927 15:01:23.978777 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:23.978826 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:23.978875 (+ 49us) server_negotiation.cc:407] Connection header received
0927 15:01:23.979887 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:23.979917 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:23.980365 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:23.980713 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:23.981492 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:23.982757 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:23.983842 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:23.984355 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:23.984536 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:23.986858 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:23.986868 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:23.986888 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:23.986922 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:23.988987 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:23.989651 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:23.989658 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:23.989667 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:23.989849 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:23.990145 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:23.990148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:23.990150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:23.991184 (+ 1034us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:23.991366 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:23.991831 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 15:01:23.992344 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":367,"thread_start_us":162,"threads_started":1}
I20260927 15:01:23.993937 22779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:23.979296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34643 (local address 127.0.0.1:59978)
0927 15:01:23.979743 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:23.979800 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 15:01:23.980158 (+ 358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:01:23.980580 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:01:23.980591 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:01:23.981045 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:01:23.981457 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:23.981475 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:23.982322 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:23.982325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:23.983526 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:23.983538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:23.984326 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:23.984330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:23.984537 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:23.985251 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:01:23.985270 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:01:23.987344 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:01:23.989766 (+ 2422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:23.989780 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:23.989788 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:23.990146 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:23.990772 (+ 626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:23.990776 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:23.990779 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:23.990936 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:23.991497 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:01:23.991504 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:01:23.991658 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 15:01:23.993305 (+ 1647us) client_negotiation.cc:240] Negotiation successful
0927 15:01:23.993738 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":101,"thread_start_us":138,"threads_started":1}
I20260927 15:01:23.995751 22775 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37189 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:01:23.996481 22700 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37239 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:01:23.997404 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:23.997101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37239 (local address 127.0.0.1:37506)
0927 15:01:23.997232 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:23.997300 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 15:01:23.998293 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:23.997963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37239 (local address 127.0.0.1:37520)
0927 15:01:23.998140 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:23.998199 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260927 15:01:23.998513 22700 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37239: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37239: connect: Connection refused (error 111)
Sep 27 15:01:23 dist-test-slave-pz0h krb5kdc[22619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 15:01:24.002316 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37239
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34643,127.0.98.61:37189,127.0.98.60:37239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:01:24.005008 22779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:23.996348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37189 (local address 127.0.0.1:51810)
0927 15:01:23.996477 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:23.996493 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:01:23.996597 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:01:23.996964 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:01:23.996967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:01:23.997126 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:01:23.997260 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:23.997267 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:23.998190 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:23.998194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:23.998930 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:23.998940 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:23.999515 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:23.999518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:23.999602 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:24.000124 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:01:24.000137 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:01:24.001661 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:01:24.003543 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:24.003546 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:24.003549 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:24.003750 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:24.004000 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:24.004003 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:24.004005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:24.004077 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:24.004469 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:01:24.004472 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:01:24.004553 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 15:01:24.004649 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 15:01:24.004819 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 15:01:24.005010 22778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:23.996496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51810 (local address 127.0.98.61:37189)
0927 15:01:23.996589 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:23.996594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:23.996609 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:01:23.996755 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:23.996759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:23.996823 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:23.996914 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:23.997379 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:23.998062 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:23.999064 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:23.999379 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:23.999472 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:24.001807 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:24.001811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:24.001814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:24.001847 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:24.003428 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:24.003840 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:24.003844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:24.003845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:24.003923 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:24.004161 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:24.004164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:24.004166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:24.004367 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:24.004470 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:24.004700 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 15:01:24.004897 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":25}
I20260927 15:01:24.006084 22775 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37239 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:01:24.006963 22779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:24.006588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37239 (local address 127.0.0.1:37536)
0927 15:01:24.006752 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:24.006893 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260927 15:01:24.007185 22723 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37239: connect: Connection refused (error 111)
I20260927 15:01:24.008090 22779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:24.007876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37239 (local address 127.0.0.1:37550)
0927 15:01:24.007983 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:24.008029 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 15:01:24.008224 22775 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37239: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37239: connect: Connection refused (error 111)
I20260927 15:01:24.029717 22700 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37239 } attempt: 1
I20260927 15:01:24.030756 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:24.030406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37239 (local address 127.0.0.1:37562)
0927 15:01:24.030572 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:24.030645 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 15:01:24.031493 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:24.031276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37239 (local address 127.0.0.1:37578)
0927 15:01:24.031379 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:24.031431 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 15:01:24.031607 22700 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37239: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37239: connect: Connection refused (error 111)
I20260927 15:01:24.057523 22775 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37239 } attempt: 1
I20260927 15:01:24.058459 22779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:24.058157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37239 (local address 127.0.0.1:37586)
0927 15:01:24.058313 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:24.058376 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 15:01:24.059424 22779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:24.059126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37239 (local address 127.0.0.1:37596)
0927 15:01:24.059300 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:24.059352 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260927 15:01:24.059644 22775 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37239: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37239: connect: Connection refused (error 111)
I20260927 15:01:24.106832 22700 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37239 } attempt: 2
I20260927 15:01:24.107816 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:24.107448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37239 (local address 127.0.0.1:37600)
0927 15:01:24.107630 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:24.107718 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 15:01:24.108633 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:24.108376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37239 (local address 127.0.0.1:37602)
0927 15:01:24.108504 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:24.108560 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 15:01:24.108865 22700 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37239: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37239: connect: Connection refused (error 111)
W20260927 15:01:24.125065 22781 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:01:24.125553 22781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:01:24.125694 22781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:01:24.125788 22781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 15:01:24.127820 22775 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37239 } attempt: 2
I20260927 15:01:24.128700 22779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:24.128395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37239 (local address 127.0.0.1:37608)
0927 15:01:24.128553 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:24.128618 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 15:01:24.129380 22779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:24.129173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37239 (local address 127.0.0.1:37622)
0927 15:01:24.129278 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:24.129323 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 15:01:24.129489 22775 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37239: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37239: connect: Connection refused (error 111)
W20260927 15:01:24.130780 22781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:01:24.130879 22781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:01:24.130916 22781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:01:24.130947 22781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:01:24.130976 22781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:01:24.136960 22781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34643,127.0.98.61:37189,127.0.98.60:37239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:01:24.138273 22781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:01:24.139606 22781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:01:24.148480 22790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:24.148480 22788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:01:24.148590 22781 server_base.cc:1034] running on GCE node
W20260927 15:01:24.148484 22787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:01:24.149451 22781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:01:24.150758 22781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:01:24.152000 22781 hybrid_clock.cc:648] HybridClock initialized: now 1790521284151974 us; error 49 us; skew 500 ppm
Sep 27 15:01:24 dist-test-slave-pz0h krb5kdc[22619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:01:24.155711 22781 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 15:01:24.156774 22781 webserver.cc:460] Webserver started at http://127.0.98.60:41301/ using document root <none> and password file <none>
I20260927 15:01:24.157387 22781 fs_manager.cc:359] Metadata directory not provided
I20260927 15:01:24.157454 22781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:01:24.157635 22781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:01:24.159986 22781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-2/data/instance:
uuid: "68e68b422dbd4c80bdf0a4d97d15f835"
format_stamp: "Formatted at 2026-09-27 15:01:24 on dist-test-slave-pz0h"
server_key: "ac49d9e4450e0f29f01f101baee64b5d"
server_key_iv: "2eb1d7623273ddb82a6b83be45411b2d"
server_key_version: "encryptionkey@0"
I20260927 15:01:24.160550 22781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-2/wal/instance:
uuid: "68e68b422dbd4c80bdf0a4d97d15f835"
format_stamp: "Formatted at 2026-09-27 15:01:24 on dist-test-slave-pz0h"
server_key: "ac49d9e4450e0f29f01f101baee64b5d"
server_key_iv: "2eb1d7623273ddb82a6b83be45411b2d"
server_key_version: "encryptionkey@0"
I20260927 15:01:24.164630 22781 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:01:24.167382 22797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:01:24.168753 22781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:01:24.168884 22781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-2/wal
uuid: "68e68b422dbd4c80bdf0a4d97d15f835"
format_stamp: "Formatted at 2026-09-27 15:01:24 on dist-test-slave-pz0h"
server_key: "ac49d9e4450e0f29f01f101baee64b5d"
server_key_iv: "2eb1d7623273ddb82a6b83be45411b2d"
server_key_version: "encryptionkey@0"
I20260927 15:01:24.169000 22781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:01:24.182687 22781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:01:24.186123 22781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:01:24.186306 22781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:01:24.193804 22781 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37239
I20260927 15:01:24.194092 22849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37239 every 8 connection(s)
I20260927 15:01:24.194885 22781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-2/data/info.pb
I20260927 15:01:24.197995 22850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:01:24.200649 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 22781
I20260927 15:01:24.200762 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790521004237705-392-0/minicluster-data/master-2/wal/instance
I20260927 15:01:24.201051 392 external_mini_cluster.cc:1428] Setting key 8663f3ce6f242503da353a3184cc6177
I20260927 15:01:24.203521 22850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34643 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:01:24.208055 22700 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37239 } attempt: 3
Sep 27 15:01:24 dist-test-slave-pz0h krb5kdc[22619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 15:01:24 dist-test-slave-pz0h krb5kdc[22619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 15:01:24 dist-test-slave-pz0h krb5kdc[22619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 15:01:24.216646 22704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:24.206080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59992 (local address 127.0.98.62:34643)
0927 15:01:24.206206 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:24.206212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:24.206511 (+ 299us) server_negotiation.cc:407] Connection header received
0927 15:01:24.206645 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:24.206648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:24.206695 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:24.206761 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:24.207595 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.208162 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.209382 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.209715 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.209828 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:24.212514 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:24.212518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:24.212521 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:24.212543 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:24.214074 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:24.215049 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:24.215052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:24.215054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:24.215140 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:24.215426 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:24.215429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:24.215431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:24.215688 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:24.215786 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:24.216306 (+ 520us) server_negotiation.cc:299] Negotiation successful
0927 15:01:24.216451 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":47}
I20260927 15:01:24.217829 22854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:24.205999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34643 (local address 127.0.0.1:59992)
0927 15:01:24.206390 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:24.206431 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:01:24.206557 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:01:24.206864 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:01:24.206870 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:01:24.207283 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:01:24.207448 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:24.207456 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.208354 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.208358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:24.209287 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:24.209294 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.209898 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.209902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:24.210116 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:24.210657 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:01:24.210673 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:01:24.212392 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:01:24.214578 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:24.214591 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:24.214605 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:24.214954 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:24.215242 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:24.215244 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:24.215246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:24.215324 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:24.215850 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:01:24.215857 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:01:24.216018 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 15:01:24.216210 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 15:01:24.216577 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":57,"spinlock_wait_cycles":6912,"thread_start_us":135,"threads_started":1}
I20260927 15:01:24.217829 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:24.202608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37624 (local address 127.0.98.60:37239)
0927 15:01:24.203094 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:24.203111 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:24.203148 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:01:24.203897 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:24.203919 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:24.204299 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:24.204632 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:24.205612 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.207064 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.208076 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.208767 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.208937 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:24.211790 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:24.211797 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:24.211810 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:24.211834 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:24.214062 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:24.214579 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:24.214591 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:24.214600 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:24.214788 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:24.215137 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:24.215140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:24.215142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:24.215792 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:24.216041 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:24.216978 (+ 937us) server_negotiation.cc:299] Negotiation successful
0927 15:01:24.217138 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":176,"threads_started":1}
I20260927 15:01:24.218319 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:24.208511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37239 (local address 127.0.0.1:37630)
0927 15:01:24.208696 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:24.208710 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:01:24.208786 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:01:24.209207 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:01:24.209210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:01:24.209710 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:01:24.209821 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:24.209827 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.210981 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.210984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:24.211794 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:24.211808 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.212369 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.212373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:24.212466 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:24.212831 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:01:24.212847 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:01:24.214163 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:01:24.216874 (+ 2711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:24.216877 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:24.216879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:24.217115 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:24.217381 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:24.217384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:24.217386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:24.217462 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:24.217832 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:01:24.217835 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:01:24.217930 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 15:01:24.218031 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 15:01:24.218175 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260927 15:01:24.218864 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:24.208578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37630 (local address 127.0.98.60:37239)
0927 15:01:24.208985 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:24.208993 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:24.209005 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:01:24.209035 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:24.209037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:24.209076 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:24.209156 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:24.209938 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.210865 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.211917 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.212253 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.212346 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:24.214289 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:24.214292 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:24.214294 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:24.214315 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:24.216740 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:24.217213 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:24.217223 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:24.217225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:24.217291 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:24.217546 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:24.217549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:24.217551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:24.217740 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:24.217825 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:24.218520 (+ 695us) server_negotiation.cc:299] Negotiation successful
0927 15:01:24.218634 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":341,"thread_start_us":132,"threads_started":1}
I20260927 15:01:24.219971 22850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37189 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:01:24.220849 22775 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37239 } attempt: 3
I20260927 15:01:24.222344 22700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1524b2a471f3404c8429b410474a3375: Bootstrap starting.
I20260927 15:01:24.225000 22700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1524b2a471f3404c8429b410474a3375: Neither blocks nor log segments found. Creating new log.
Sep 27 15:01:24 dist-test-slave-pz0h krb5kdc[22619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 15:01:24.225939 22700 log.cc:824] T 00000000000000000000000000000000 P 1524b2a471f3404c8429b410474a3375: Log is configured to *not* fsync() on all Append() calls
I20260927 15:01:24.228655 22700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1524b2a471f3404c8429b410474a3375: No bootstrap required, opened a new log
Sep 27 15:01:24 dist-test-slave-pz0h krb5kdc[22619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:01:24.229981 22778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:24.220783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51822 (local address 127.0.98.61:37189)
0927 15:01:24.220893 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:24.220896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:24.220907 (+ 11us) server_negotiation.cc:407] Connection header received
0927 15:01:24.220943 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:24.220946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:24.220991 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:24.221061 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:24.221462 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.221983 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.222774 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.223107 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.223197 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:24.225581 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:24.225586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:24.225589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:24.225617 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:24.227939 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:24.228575 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:24.228579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:24.228581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:24.228665 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:24.228976 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:24.228979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:24.228981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:24.229218 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:24.229294 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:24.229676 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 15:01:24.229806 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 15:01:24.230508 22854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:24.220460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37189 (local address 127.0.0.1:51822)
0927 15:01:24.220627 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:24.220641 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:01:24.220718 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:01:24.221079 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:01:24.221082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:01:24.221253 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:01:24.221352 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:24.221358 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.222089 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.222093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:24.222664 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:24.222673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.223315 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.223319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:24.223423 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:24.223950 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:01:24.223966 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:01:24.225425 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:01:24.228073 (+ 2648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:24.228077 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:24.228079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:24.228462 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:24.228779 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:24.228782 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:24.228784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:24.228876 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:24.229364 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:01:24.229367 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:01:24.229471 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 15:01:24.230202 (+ 731us) client_negotiation.cc:240] Negotiation successful
0927 15:01:24.230359 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
Sep 27 15:01:24 dist-test-slave-pz0h krb5kdc[22619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 15:01:24.231384 22850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37239 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:01:24.233295 22700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1524b2a471f3404c8429b410474a3375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1524b2a471f3404c8429b410474a3375" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34643 } } peers { permanent_uuid: "075139e2a39e43109553a940df31de5b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37189 } } peers { permanent_uuid: "68e68b422dbd4c80bdf0a4d97d15f835" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37239 } }
I20260927 15:01:24.233585 22700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1524b2a471f3404c8429b410474a3375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1524b2a471f3404c8429b410474a3375, State: Initialized, Role: FOLLOWER
I20260927 15:01:24.233992 22700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1524b2a471f3404c8429b410474a3375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1524b2a471f3404c8429b410474a3375" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34643 } } peers { permanent_uuid: "075139e2a39e43109553a940df31de5b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37189 } } peers { permanent_uuid: "68e68b422dbd4c80bdf0a4d97d15f835" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37239 } }
I20260927 15:01:24.234887 22704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:24.221116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59994 (local address 127.0.98.62:34643)
0927 15:01:24.223404 (+ 2288us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:24.223409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:24.223422 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:01:24.223456 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:24.223459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:24.223507 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:24.223582 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:24.224449 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.224938 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.225824 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.226428 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.226533 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:24.230373 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:24.230376 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:24.230379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:24.230407 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:24.232524 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:24.233087 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:24.233091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:24.233093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:24.233199 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:24.233503 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:24.233506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:24.233508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:24.233880 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:24.234046 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:24.234575 (+ 529us) server_negotiation.cc:299] Negotiation successful
0927 15:01:24.234707 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2207}
I20260927 15:01:24.235044 22779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:24.223033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37239 (local address 127.0.0.1:37636)
0927 15:01:24.223165 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:24.223181 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:01:24.223263 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:01:24.226777 (+ 3514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:01:24.226780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:01:24.227000 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:01:24.227112 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:24.227119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.228105 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.228108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:24.228831 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:24.228841 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.229444 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.229448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:24.229532 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:24.230003 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:01:24.230017 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:01:24.231537 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:01:24.233416 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:24.233419 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:24.233421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:24.233687 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:24.233989 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:24.233992 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:24.233993 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:24.234081 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:24.234522 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:01:24.234526 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:01:24.234641 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 15:01:24.234721 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 15:01:24.234897 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 15:01:24.235757 22700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1524b2a471f3404c8429b410474a3375 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:01:24.236469 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:24.223274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37636 (local address 127.0.98.60:37239)
0927 15:01:24.226505 (+ 3231us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:24.226509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:24.226523 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:01:24.226556 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:24.226558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:24.226601 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:24.226728 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:24.227268 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.227972 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.228975 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.229312 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.229760 (+ 448us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:24.231703 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:24.231706 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:24.231710 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:24.231734 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:24.233277 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:24.233808 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:24.233811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:24.233813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:24.233895 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:24.234195 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:24.234198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:24.234199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:24.234412 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:24.235358 (+ 946us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:24.235517 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 15:01:24.235658 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3145}
Sep 27 15:01:24 dist-test-slave-pz0h krb5kdc[22619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 15:01:24.240229 22775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 075139e2a39e43109553a940df31de5b: Bootstrap starting.
I20260927 15:01:24.241593 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:24.232258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37648 (local address 127.0.98.60:37239)
0927 15:01:24.232370 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:24.232374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:24.232622 (+ 248us) server_negotiation.cc:407] Connection header received
0927 15:01:24.232647 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:24.232650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:24.232697 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:24.232754 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:24.233213 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.233965 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.234939 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.235300 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.235392 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:24.237954 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:24.237959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:24.237961 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:24.237986 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:24.239670 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:24.240110 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:24.240113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:24.240115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:24.240187 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:24.240534 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:24.240537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:24.240539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:24.240883 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:24.240968 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:24.241303 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 15:01:24.241435 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 15:01:24.242416 22854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:24.231897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37239 (local address 127.0.0.1:37648)
0927 15:01:24.232454 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:24.232467 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:01:24.232544 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:01:24.232859 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:01:24.232862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:01:24.233018 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:01:24.233112 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:24.233119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.234064 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.234066 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:24.234828 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:24.234838 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.235499 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.235502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:24.235580 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:24.236031 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:01:24.236042 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:01:24.237813 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:01:24.239791 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:24.239793 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:24.239796 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:24.240018 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:24.240292 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:24.240294 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:24.240296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:24.240450 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:24.241028 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:01:24.241031 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:01:24.241121 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 15:01:24.242090 (+ 969us) client_negotiation.cc:240] Negotiation successful
0927 15:01:24.242259 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":131,"threads_started":1}
Sep 27 15:01:24 dist-test-slave-pz0h krb5kdc[22619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 15:01:24.244874 22775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 075139e2a39e43109553a940df31de5b: Neither blocks nor log segments found. Creating new log.
I20260927 15:01:24.241727 22632 master_runner.cc:186] Error getting master registration for 127.0.98.61:37189: OK, instance_id { permanent_uuid: "075139e2a39e43109553a940df31de5b" instance_seqno: 1790521283964500 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:01:24.245966 22775 log.cc:824] T 00000000000000000000000000000000 P 075139e2a39e43109553a940df31de5b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:01:24.246071 22632 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:01:24.246518 22858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1524b2a471f3404c8429b410474a3375 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1524b2a471f3404c8429b410474a3375" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34643 } } peers { permanent_uuid: "075139e2a39e43109553a940df31de5b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37189 } } peers { permanent_uuid: "68e68b422dbd4c80bdf0a4d97d15f835" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37239 } } }
I20260927 15:01:24.246652 22858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1524b2a471f3404c8429b410474a3375 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:01:24.246701 22850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68e68b422dbd4c80bdf0a4d97d15f835: Bootstrap starting.
I20260927 15:01:24.248415 22775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 075139e2a39e43109553a940df31de5b: No bootstrap required, opened a new log
I20260927 15:01:24.249632 22850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68e68b422dbd4c80bdf0a4d97d15f835: Neither blocks nor log segments found. Creating new log.
I20260927 15:01:24.250324 22778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:24.237674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51832 (local address 127.0.98.61:37189)
0927 15:01:24.237789 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:24.237793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:24.237952 (+ 159us) server_negotiation.cc:407] Connection header received
0927 15:01:24.238144 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:24.238148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:24.238198 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:24.238252 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:24.239315 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.240035 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.241054 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.241380 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.241451 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:24.244653 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:24.244658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:24.244661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:24.244700 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:24.246447 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:24.248153 (+ 1706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:24.248158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:24.248159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:24.248303 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:24.248687 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:24.248690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:24.248692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:24.248927 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:24.249109 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:24.249861 (+ 752us) server_negotiation.cc:299] Negotiation successful
0927 15:01:24.250118 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:01:24.250526 22850 log.cc:824] T 00000000000000000000000000000000 P 68e68b422dbd4c80bdf0a4d97d15f835: Log is configured to *not* fsync() on all Append() calls
W20260927 15:01:24.252139 22869 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1524b2a471f3404c8429b410474a3375: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:01:24.252238 22869 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:01:24.252460 22775 raft_consensus.cc:348] T 00000000000000000000000000000000 P 075139e2a39e43109553a940df31de5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1524b2a471f3404c8429b410474a3375" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34643 } } peers { permanent_uuid: "075139e2a39e43109553a940df31de5b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37189 } } peers { permanent_uuid: "68e68b422dbd4c80bdf0a4d97d15f835" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37239 } }
I20260927 15:01:24.252740 22775 raft_consensus.cc:729] T 00000000000000000000000000000000 P 075139e2a39e43109553a940df31de5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075139e2a39e43109553a940df31de5b, State: Initialized, Role: FOLLOWER
I20260927 15:01:24.253098 22775 consensus_queue.cc:260] T 00000000000000000000000000000000 P 075139e2a39e43109553a940df31de5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1524b2a471f3404c8429b410474a3375" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34643 } } peers { permanent_uuid: "075139e2a39e43109553a940df31de5b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37189 } } peers { permanent_uuid: "68e68b422dbd4c80bdf0a4d97d15f835" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37239 } }
I20260927 15:01:24.253847 22873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 075139e2a39e43109553a940df31de5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1524b2a471f3404c8429b410474a3375" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34643 } } peers { permanent_uuid: "075139e2a39e43109553a940df31de5b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37189 } } peers { permanent_uuid: "68e68b422dbd4c80bdf0a4d97d15f835" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37239 } } }
I20260927 15:01:24.253995 22873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 075139e2a39e43109553a940df31de5b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:01:24.254582 22775 sys_catalog.cc:564] T 00000000000000000000000000000000 P 075139e2a39e43109553a940df31de5b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:01:24.255504 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:01:24.257042 22674 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1524b2a471f3404c8429b410474a3375"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59994
I20260927 15:01:24.257340 22674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1524b2a471f3404c8429b410474a3375 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 15:01:24.257467 22674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1524b2a471f3404c8429b410474a3375 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:01:24.258948 22674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1524b2a471f3404c8429b410474a3375 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1524b2a471f3404c8429b410474a3375" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34643 } } peers { permanent_uuid: "075139e2a39e43109553a940df31de5b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37189 } } peers { permanent_uuid: "68e68b422dbd4c80bdf0a4d97d15f835" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37239 } }
I20260927 15:01:24.259109 22850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68e68b422dbd4c80bdf0a4d97d15f835: No bootstrap required, opened a new log
I20260927 15:01:24.260788 22749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1524b2a471f3404c8429b410474a3375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "075139e2a39e43109553a940df31de5b"
I20260927 15:01:24.262564 22850 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68e68b422dbd4c80bdf0a4d97d15f835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1524b2a471f3404c8429b410474a3375" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34643 } } peers { permanent_uuid: "075139e2a39e43109553a940df31de5b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37189 } } peers { permanent_uuid: "68e68b422dbd4c80bdf0a4d97d15f835" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37239 } }
I20260927 15:01:24.262892 22850 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68e68b422dbd4c80bdf0a4d97d15f835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68e68b422dbd4c80bdf0a4d97d15f835, State: Initialized, Role: FOLLOWER
I20260927 15:01:24.263167 22674 leader_election.cc:288] T 00000000000000000000000000000000 P 1524b2a471f3404c8429b410474a3375 [CANDIDATE]: Term 1 election: Requested vote from peers 075139e2a39e43109553a940df31de5b (127.0.98.61:37189), 68e68b422dbd4c80bdf0a4d97d15f835 (127.0.98.60:37239)
I20260927 15:01:24.263271 22850 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68e68b422dbd4c80bdf0a4d97d15f835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1524b2a471f3404c8429b410474a3375" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34643 } } peers { permanent_uuid: "075139e2a39e43109553a940df31de5b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37189 } } peers { permanent_uuid: "68e68b422dbd4c80bdf0a4d97d15f835" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37239 } }
I20260927 15:01:24.264392 22749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 075139e2a39e43109553a940df31de5b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:01:24.265914 22749 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 075139e2a39e43109553a940df31de5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1524b2a471f3404c8429b410474a3375 in term 1.
I20260927 15:01:24.265923 22889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68e68b422dbd4c80bdf0a4d97d15f835 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1524b2a471f3404c8429b410474a3375" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34643 } } peers { permanent_uuid: "075139e2a39e43109553a940df31de5b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37189 } } peers { permanent_uuid: "68e68b422dbd4c80bdf0a4d97d15f835" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37239 } } }
I20260927 15:01:24.266619 22850 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68e68b422dbd4c80bdf0a4d97d15f835 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:01:24.266687 22824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1524b2a471f3404c8429b410474a3375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "68e68b422dbd4c80bdf0a4d97d15f835"
I20260927 15:01:24.266947 22824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68e68b422dbd4c80bdf0a4d97d15f835 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:01:24.268337 22824 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 68e68b422dbd4c80bdf0a4d97d15f835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1524b2a471f3404c8429b410474a3375 in term 1.
I20260927 15:01:24.268847 22648 leader_election.cc:302] T 00000000000000000000000000000000 P 1524b2a471f3404c8429b410474a3375 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1524b2a471f3404c8429b410474a3375, 68e68b422dbd4c80bdf0a4d97d15f835; no voters:
I20260927 15:01:24.269207 22858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1524b2a471f3404c8429b410474a3375 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:01:24.269483 22858 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1524b2a471f3404c8429b410474a3375 [term 1 LEADER]: Becoming Leader. State: Replica: 1524b2a471f3404c8429b410474a3375, State: Running, Role: LEADER
I20260927 15:01:24.269876 22858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1524b2a471f3404c8429b410474a3375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1524b2a471f3404c8429b410474a3375" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34643 } } peers { permanent_uuid: "075139e2a39e43109553a940df31de5b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37189 } } peers { permanent_uuid: "68e68b422dbd4c80bdf0a4d97d15f835" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37239 } }
I20260927 15:01:24.271955 22858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1524b2a471f3404c8429b410474a3375 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1524b2a471f3404c8429b410474a3375. Latest consensus state: current_term: 1 leader_uuid: "1524b2a471f3404c8429b410474a3375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1524b2a471f3404c8429b410474a3375" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34643 } } peers { permanent_uuid: "075139e2a39e43109553a940df31de5b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37189 } } peers { permanent_uuid: "68e68b422dbd4c80bdf0a4d97d15f835" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37239 } } }
I20260927 15:01:24.272094 22858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1524b2a471f3404c8429b410474a3375 [sys.catalog]: This master's current role is: LEADER
W20260927 15:01:24.272275 22884 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 075139e2a39e43109553a940df31de5b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:01:24.272367 22884 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260927 15:01:24.272217 22900 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 68e68b422dbd4c80bdf0a4d97d15f835: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:01:24.272938 22900 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:01:24.268381 22707 master_runner.cc:186] Error getting master registration for 127.0.98.60:37239: OK, instance_id { permanent_uuid: "68e68b422dbd4c80bdf0a4d97d15f835" instance_seqno: 1790521284191666 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:01:24.273216 22707 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:01:24.273298 22889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68e68b422dbd4c80bdf0a4d97d15f835 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:01:24.278023 22905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:01:24.279745 22905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:01:24.286870 22704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:24.275384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59998 (local address 127.0.98.62:34643)
0927 15:01:24.275494 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:24.275498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:24.276894 (+ 1396us) server_negotiation.cc:407] Connection header received
0927 15:01:24.277070 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:24.277073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:24.277122 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:24.277172 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:24.278153 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.278960 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.280133 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.280448 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.280519 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:24.282445 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:24.282450 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:24.282453 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:24.282484 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:24.284773 (+ 2289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:24.285345 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:24.285349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:24.285351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:24.285444 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:24.285778 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:24.285782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:24.285784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:24.286022 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:24.286130 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:24.286498 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 15:01:24.286633 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 15:01:24.293777 22778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:24.275744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51844 (local address 127.0.98.61:37189)
0927 15:01:24.275851 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:24.275855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:24.279091 (+ 3236us) server_negotiation.cc:407] Connection header received
0927 15:01:24.279273 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:24.279276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:24.279324 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:24.279372 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:24.283263 (+ 3891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.284337 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.287114 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.287486 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.287565 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:24.289948 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:24.289953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:24.289955 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:24.289990 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:24.291757 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:24.292295 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:24.292299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:24.292301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:24.292391 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:24.292700 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:24.292704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:24.292706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:24.292946 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:24.293073 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:24.293433 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 15:01:24.293581 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 15:01:24.298349 22824 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 68e68b422dbd4c80bdf0a4d97d15f835 [term 1 FOLLOWER]: Refusing update from remote peer 1524b2a471f3404c8429b410474a3375: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:01:24.298945 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:24.276097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37650 (local address 127.0.98.60:37239)
0927 15:01:24.276216 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:24.276220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:24.285837 (+ 9617us) server_negotiation.cc:407] Connection header received
0927 15:01:24.286020 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:24.286024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:24.286077 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:24.286130 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:24.288626 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.289396 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.290557 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:24.290957 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:24.291040 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:24.295147 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:24.295151 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:24.295154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:24.295188 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:24.296971 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:24.297493 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:24.297497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:24.297499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:24.297588 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:24.297911 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:24.297915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:24.297916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:24.298146 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:24.298268 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:24.298612 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 15:01:24.298755 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 15:01:24.299119 22858 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1524b2a471f3404c8429b410474a3375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68e68b422dbd4c80bdf0a4d97d15f835" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:01:24.300870 22749 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 075139e2a39e43109553a940df31de5b [term 1 FOLLOWER]: Refusing update from remote peer 1524b2a471f3404c8429b410474a3375: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:01:24.301669 22858 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1524b2a471f3404c8429b410474a3375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075139e2a39e43109553a940df31de5b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:01:24.304412 22889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68e68b422dbd4c80bdf0a4d97d15f835 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1524b2a471f3404c8429b410474a3375. Latest consensus state: current_term: 1 leader_uuid: "1524b2a471f3404c8429b410474a3375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1524b2a471f3404c8429b410474a3375" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34643 } } peers { permanent_uuid: "075139e2a39e43109553a940df31de5b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37189 } } peers { permanent_uuid: "68e68b422dbd4c80bdf0a4d97d15f835" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37239 } } }
I20260927 15:01:24.304553 22889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68e68b422dbd4c80bdf0a4d97d15f835 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:01:24.307642 22873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 075139e2a39e43109553a940df31de5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1524b2a471f3404c8429b410474a3375. Latest consensus state: current_term: 1 leader_uuid: "1524b2a471f3404c8429b410474a3375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1524b2a471f3404c8429b410474a3375" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34643 } } peers { permanent_uuid: "075139e2a39e43109553a940df31de5b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37189 } } peers { permanent_uuid: "68e68b422dbd4c80bdf0a4d97d15f835" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37239 } } }
I20260927 15:01:24.307816 22873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 075139e2a39e43109553a940df31de5b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:01:24.311985 22901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1524b2a471f3404c8429b410474a3375 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1524b2a471f3404c8429b410474a3375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1524b2a471f3404c8429b410474a3375" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34643 } } peers { permanent_uuid: "075139e2a39e43109553a940df31de5b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37189 } } peers { permanent_uuid: "68e68b422dbd4c80bdf0a4d97d15f835" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37239 } } }
I20260927 15:01:24.312175 22901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1524b2a471f3404c8429b410474a3375 [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:24.312309 22909 mvcc.cc:205] Tried to move back new op lower bound from 7333975180476928000 to 7333975180373667840. Current Snapshot: MvccSnapshot[applied={T|T < 7333975180476928000}]
I20260927 15:01:24.314066 22873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 075139e2a39e43109553a940df31de5b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1524b2a471f3404c8429b410474a3375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1524b2a471f3404c8429b410474a3375" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34643 } } peers { permanent_uuid: "075139e2a39e43109553a940df31de5b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37189 } } peers { permanent_uuid: "68e68b422dbd4c80bdf0a4d97d15f835" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37239 } } }
I20260927 15:01:24.314232 22873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 075139e2a39e43109553a940df31de5b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:01:24.318540 22912 mvcc.cc:205] Tried to move back new op lower bound from 7333975180476928000 to 7333975180373667840. Current Snapshot: MvccSnapshot[applied={T|T < 7333975180476928000}]
I20260927 15:01:24.322332 22858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1524b2a471f3404c8429b410474a3375 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1524b2a471f3404c8429b410474a3375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1524b2a471f3404c8429b410474a3375" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34643 } } peers { permanent_uuid: "075139e2a39e43109553a940df31de5b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37189 } } peers { permanent_uuid: "68e68b422dbd4c80bdf0a4d97d15f835" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37239 } } }
I20260927 15:01:24.322496 22858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1524b2a471f3404c8429b410474a3375 [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:24.324602 22889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68e68b422dbd4c80bdf0a4d97d15f835 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1524b2a471f3404c8429b410474a3375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1524b2a471f3404c8429b410474a3375" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34643 } } peers { permanent_uuid: "075139e2a39e43109553a940df31de5b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37189 } } peers { permanent_uuid: "68e68b422dbd4c80bdf0a4d97d15f835" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37239 } } }
I20260927 15:01:24.324764 22889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68e68b422dbd4c80bdf0a4d97d15f835 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:01:24.325912 22910 mvcc.cc:205] Tried to move back new op lower bound from 7333975180476928000 to 7333975180373667840. Current Snapshot: MvccSnapshot[applied={T|T < 7333975180476928000}]
I20260927 15:01:24.332144 22905 catalog_manager.cc:1347] Generated new cluster ID: 1b409fd730a84362bab89406216e7076
I20260927 15:01:24.332254 22905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:01:24.374421 22905 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:01:24.377508 22905 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:01:24.395448 22905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1524b2a471f3404c8429b410474a3375: Generated new TSK 0
I20260927 15:01:24.396409 22905 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:01:24.432920 22704 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:01:24.429101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60002 (local address 127.0.98.62:34643)
0927 15:01:24.429269 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:24.429275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:24.431299 (+ 2024us) server_negotiation.cc:407] Connection header received
0927 15:01:24.431848 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:24.431852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:24.431915 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:24.431978 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:24.432757 (+ 779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60002: BlockingRecv error: recv got EOF from 127.0.0.1:60002 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 15:01:24.434443 22778 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:01:24.430062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51856 (local address 127.0.98.61:37189)
0927 15:01:24.430212 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:24.430217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:24.433361 (+ 3144us) server_negotiation.cc:407] Connection header received
0927 15:01:24.433562 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:24.433566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:24.433623 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:24.433683 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:24.434303 (+ 620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51856: BlockingRecv error: recv got EOF from 127.0.0.1:51856 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 15:01:24.436306 22855 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:01:24.430969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37662 (local address 127.0.98.60:37239)
0927 15:01:24.431122 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:24.431127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:24.435056 (+ 3929us) server_negotiation.cc:407] Connection header received
0927 15:01:24.435487 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:24.435490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:24.435547 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:24.435603 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:24.436176 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37662: BlockingRecv error: recv got EOF from 127.0.0.1:37662 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 15:01:24.449055 22778 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:01:24.447374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51866 (local address 127.0.98.61:37189)
0927 15:01:24.447510 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:24.447515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:24.447860 (+ 345us) server_negotiation.cc:407] Connection header received
0927 15:01:24.448107 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:24.448110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:24.448212 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:24.448270 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:24.448921 (+ 651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51866: BlockingRecv error: recv got EOF from 127.0.0.1:51866 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 15:01:24.460202 22778 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:01:24.453551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51878 (local address 127.0.98.61:37189)
0927 15:01:24.453688 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:24.453693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:24.459181 (+ 5488us) server_negotiation.cc:407] Connection header received
0927 15:01:24.459337 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:24.459342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:24.459406 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:24.459481 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:24.460056 (+ 575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51878: BlockingRecv error: recv got EOF from 127.0.0.1:51878 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 15:01:24.468706 22778 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:01:24.467358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51892 (local address 127.0.98.61:37189)
0927 15:01:24.467502 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:24.467506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:24.467814 (+ 308us) server_negotiation.cc:407] Connection header received
0927 15:01:24.468000 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:24.468003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:24.468057 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:24.468131 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:24.468594 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51892: BlockingRecv error: recv got EOF from 127.0.0.1:51892 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 15:01:24.476811 22778 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:01:24.475884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51900 (local address 127.0.98.61:37189)
0927 15:01:24.476051 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:24.476055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:24.476069 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:01:24.476121 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:24.476123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:24.476165 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:24.476233 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:24.476694 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51900: BlockingRecv error: recv got EOF from 127.0.0.1:51900 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260927 15:01:24.488497 22778 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:01:24.487295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51910 (local address 127.0.98.61:37189)
0927 15:01:24.487422 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:24.487427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:24.487660 (+ 233us) server_negotiation.cc:407] Connection header received
0927 15:01:24.487810 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:24.487813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:24.487862 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:24.487928 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:24.488368 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51910: BlockingRecv error: recv got EOF from 127.0.0.1:51910 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 15:01:24.508001 22778 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:01:24.506611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51914 (local address 127.0.98.61:37189)
0927 15:01:24.506774 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:24.506779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:24.507013 (+ 234us) server_negotiation.cc:407] Connection header received
0927 15:01:24.507165 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:24.507168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:24.507216 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:24.507312 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:24.507873 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51914: BlockingRecv error: recv got EOF from 127.0.0.1:51914 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260927 15:01:24.543406 22778 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:01:24.542221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51916 (local address 127.0.98.61:37189)
0927 15:01:24.542403 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:24.542407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:24.542499 (+ 92us) server_negotiation.cc:407] Connection header received
0927 15:01:24.542638 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:24.542642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:24.542709 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:24.542785 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:24.543270 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51916: BlockingRecv error: recv got EOF from 127.0.0.1:51916 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260927 15:01:24.610937 22778 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:01:24.609921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51924 (local address 127.0.98.61:37189)
0927 15:01:24.610072 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:24.610077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:24.610122 (+ 45us) server_negotiation.cc:407] Connection header received
0927 15:01:24.610260 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:24.610263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:24.610310 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:24.610391 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:24.610828 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51924: BlockingRecv error: recv got EOF from 127.0.0.1:51924 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 15:01:24.742656 22778 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:01:24.741454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51936 (local address 127.0.98.61:37189)
0927 15:01:24.741571 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:24.741575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:24.741823 (+ 248us) server_negotiation.cc:407] Connection header received
0927 15:01:24.741959 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:24.741962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:24.742014 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:24.742082 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:24.742534 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51936: BlockingRecv error: recv got EOF from 127.0.0.1:51936 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 15:01:25.002106 22778 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:01:25.000916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51952 (local address 127.0.98.61:37189)
0927 15:01:25.001088 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:25.001092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:25.001183 (+ 91us) server_negotiation.cc:407] Connection header received
0927 15:01:25.001330 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:25.001333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:25.001382 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:25.001451 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:25.001988 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51952: BlockingRecv error: recv got EOF from 127.0.0.1:51952 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 15:01:25.274197 22884 catalog_manager.cc:1259] Loaded cluster ID: 1b409fd730a84362bab89406216e7076
I20260927 15:01:25.274293 22884 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 075139e2a39e43109553a940df31de5b: loading cluster ID for follower catalog manager: success
I20260927 15:01:25.274627 22900 catalog_manager.cc:1259] Loaded cluster ID: 1b409fd730a84362bab89406216e7076
I20260927 15:01:25.274713 22900 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 68e68b422dbd4c80bdf0a4d97d15f835: loading cluster ID for follower catalog manager: success
I20260927 15:01:25.276443 22884 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 075139e2a39e43109553a940df31de5b: acquiring CA information for follower catalog manager: success
I20260927 15:01:25.276860 22900 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 68e68b422dbd4c80bdf0a4d97d15f835: acquiring CA information for follower catalog manager: success
I20260927 15:01:25.277899 22884 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 075139e2a39e43109553a940df31de5b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:01:25.278167 22900 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 68e68b422dbd4c80bdf0a4d97d15f835: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:01:25.520270 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:25.516392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51962 (local address 127.0.98.61:37189)
0927 15:01:25.516627 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:25.516630 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:25.516647 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:01:25.516780 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:25.516782 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:25.516907 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:25.517017 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:01:25.517267 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:25.517806 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:25.518659 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:25.518927 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:25.519001 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:25.519338 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:01:25.519824 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:01:25.519930 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:25.520041 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 15:01:25.520103 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":70,"threads_started":1}
I20260927 15:01:25.527446 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:25.523360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37666 (local address 127.0.98.60:37239)
0927 15:01:25.523599 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:25.523602 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:25.523617 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:01:25.523757 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:25.523760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:25.523887 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:25.523987 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:01:25.524187 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:25.524711 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:25.525570 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:25.525806 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:25.525894 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:25.526184 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:01:25.526637 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:01:25.526737 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:25.527095 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 15:01:25.527226 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":84,"threads_started":1}
I20260927 15:01:25.535547 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:25.531259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37670 (local address 127.0.98.60:37239)
0927 15:01:25.531409 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:25.531412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:25.531682 (+ 270us) server_negotiation.cc:407] Connection header received
0927 15:01:25.531751 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:25.531754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:25.531815 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:25.531889 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:01:25.532278 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:25.533084 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:25.534206 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:25.534486 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:25.534592 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:25.534996 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:01:25.535098 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:01:25.535167 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:25.535304 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 15:01:25.535373 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:01:25.535965 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:25.531496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51966 (local address 127.0.98.61:37189)
0927 15:01:25.531625 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:25.531630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:25.532018 (+ 388us) server_negotiation.cc:407] Connection header received
0927 15:01:25.532085 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:25.532088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:25.532144 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:25.532205 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:01:25.532471 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:25.533012 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:25.534403 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:25.534721 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:25.534870 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:25.535215 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:01:25.535349 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:01:25.535517 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:25.535693 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 15:01:25.535789 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:01:25.536995 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:25.531113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60016 (local address 127.0.98.62:34643)
0927 15:01:25.531592 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:25.531597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:25.531899 (+ 302us) server_negotiation.cc:407] Connection header received
0927 15:01:25.532048 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:25.532051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:25.532253 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:25.532376 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:01:25.532569 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:25.533410 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:25.534649 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:25.535082 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:25.535170 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:25.535833 (+ 663us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:01:25.536391 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:01:25.536463 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:25.536684 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 15:01:25.536788 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":125,"threads_started":1}
I20260927 15:01:25.549466 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 22632
I20260927 15:01:25.555848 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 22707
I20260927 15:01:25.561556 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 22781
2026-09-27T15:01:25Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3109 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:01:25 dist-test-slave-pz0h krb5kdc[22999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:01:25 dist-test-slave-pz0h krb5kdc[22999](info): set up 2 sockets
Sep 27 15:01:25 dist-test-slave-pz0h krb5kdc[22999](info): commencing operation
krb5kdc: starting...
W20260927 15:01:26.527130 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/krb5kdc/test-user.keytab.
Sep 27 15:01:26 dist-test-slave-pz0h krb5kdc[22999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:01:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:01:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 15:01:26.649978 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46587
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46587,127.0.98.61:43565,127.0.98.60:39817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:01:26.769933 23012 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:01:26.770283 23012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:01:26.770308 23012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:01:26.770354 23012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:01:26.774664 23012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:01:26.774758 23012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:01:26.774777 23012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:01:26.774821 23012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:01:26.774838 23012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:01:26.780610 23012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46587,127.0.98.61:43565,127.0.98.60:39817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:01:26.781821 23012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:01:26.782949 23012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:01:26.791697 23018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:26.791710 23017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:26.791705 23020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:01:26.792124 23012 server_base.cc:1034] running on GCE node
I20260927 15:01:26.792708 23012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:01:26.794005 23012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:01:26.795173 23012 hybrid_clock.cc:648] HybridClock initialized: now 1790521286795152 us; error 37 us; skew 500 ppm
Sep 27 15:01:26 dist-test-slave-pz0h krb5kdc[22999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:01:26.798142 23012 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 15:01:26.799213 23012 webserver.cc:460] Webserver started at http://127.0.98.62:39551/ using document root <none> and password file <none>
I20260927 15:01:26.799782 23012 fs_manager.cc:359] Metadata directory not provided
I20260927 15:01:26.799845 23012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:01:26.800015 23012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:01:26.802169 23012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-0/data/instance:
uuid: "adc5a1b8348b4bbcaa43e1df788317c8"
format_stamp: "Formatted at 2026-09-27 15:01:26 on dist-test-slave-pz0h"
server_key: "a8ffaae1215112a0d9aa00e254a679b5"
server_key_iv: "5ca14ebecffa24b7006aa3d2ef4ebc7c"
server_key_version: "encryptionkey@0"
I20260927 15:01:26.802745 23012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-0/wal/instance:
uuid: "adc5a1b8348b4bbcaa43e1df788317c8"
format_stamp: "Formatted at 2026-09-27 15:01:26 on dist-test-slave-pz0h"
server_key: "a8ffaae1215112a0d9aa00e254a679b5"
server_key_iv: "5ca14ebecffa24b7006aa3d2ef4ebc7c"
server_key_version: "encryptionkey@0"
I20260927 15:01:26.806768 23012 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 15:01:26.809404 23027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:01:26.810719 23012 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:01:26.810906 23012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-0/wal
uuid: "adc5a1b8348b4bbcaa43e1df788317c8"
format_stamp: "Formatted at 2026-09-27 15:01:26 on dist-test-slave-pz0h"
server_key: "a8ffaae1215112a0d9aa00e254a679b5"
server_key_iv: "5ca14ebecffa24b7006aa3d2ef4ebc7c"
server_key_version: "encryptionkey@0"
I20260927 15:01:26.811025 23012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:01:26.831163 23012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:01:26.834554 23012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:01:26.834739 23012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:01:26.842257 23012 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46587
I20260927 15:01:26.842533 23079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46587 every 8 connection(s)
I20260927 15:01:26.843300 23012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-0/data/info.pb
I20260927 15:01:26.846305 23080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:01:26.847087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 23012
I20260927 15:01:26.847183 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-0/wal/instance
I20260927 15:01:26.847451 392 external_mini_cluster.cc:1428] Setting key 82d580cb0b7b388af3802ac87e8c539f
I20260927 15:01:26.850999 23080 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46587 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 15:01:26 dist-test-slave-pz0h krb5kdc[22999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:01:26.867591 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:26.853441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46587 (local address 127.0.0.1:46018)
0927 15:01:26.854003 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:26.854075 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 15:01:26.855108 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:01:26.855961 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:01:26.855968 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:01:26.856401 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:01:26.856704 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:26.856719 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:26.858311 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:26.858315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:26.859485 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:26.859501 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:26.860404 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:26.860407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:26.860500 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:26.861083 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:01:26.861101 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:01:26.862543 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:01:26.864556 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:26.864565 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:26.864577 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:26.864795 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:26.865097 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:26.865100 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:26.865102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:26.865210 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:26.865830 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:01:26.865836 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:01:26.865969 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 15:01:26.866136 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 15:01:26.866533 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":107,"thread_start_us":140,"threads_started":1}
I20260927 15:01:26.867591 23084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:26.853442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46018 (local address 127.0.98.62:46587)
0927 15:01:26.854101 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:26.854115 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:26.854240 (+ 125us) server_negotiation.cc:407] Connection header received
0927 15:01:26.855474 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:26.855481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:26.855713 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:26.855936 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:26.856938 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:26.858161 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:26.859654 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:26.860245 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:26.860441 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:26.862704 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:26.862710 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:26.862723 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:26.862745 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:26.864413 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:26.864898 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:26.864904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:26.864905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:26.865011 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:26.865292 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:26.865295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:26.865296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:26.865722 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:26.865850 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:26.866210 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 15:01:26.866466 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":526,"spinlock_wait_cycles":59136,"thread_start_us":192,"threads_started":1}
I20260927 15:01:26.870040 23080 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43565 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:01:26.871006 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:26.870583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43565 (local address 127.0.0.1:48332)
0927 15:01:26.870769 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:26.870906 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260927 15:01:26.871223 23028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43565: connect: Connection refused (error 111)
I20260927 15:01:26.872110 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:26.871898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43565 (local address 127.0.0.1:48334)
0927 15:01:26.871997 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:26.872042 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 15:01:26.872243 23080 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43565: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43565: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 15:01:26.873939 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43565
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46587,127.0.98.61:43565,127.0.98.60:39817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:01:26.921511 23080 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43565 } attempt: 1
I20260927 15:01:26.922420 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:26.922062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43565 (local address 127.0.0.1:48342)
0927 15:01:26.922238 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:26.922318 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 15:01:26.923223 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:26.922981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43565 (local address 127.0.0.1:48348)
0927 15:01:26.923096 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:26.923154 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 15:01:26.923357 23080 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43565: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43565: connect: Connection refused (error 111)
I20260927 15:01:26.991505 23080 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43565 } attempt: 2
I20260927 15:01:26.992439 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:26.992084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43565 (local address 127.0.0.1:48354)
0927 15:01:26.992269 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:26.992345 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 15:01:26.993296 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:26.993057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43565 (local address 127.0.0.1:48360)
0927 15:01:26.993164 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:26.993219 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 15:01:26.993420 23080 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43565: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43565: connect: Connection refused (error 111)
W20260927 15:01:26.995625 23087 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:01:26.995932 23087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:01:26.995956 23087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:01:26.996013 23087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:01:27.000342 23087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:01:27.000420 23087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:01:27.000439 23087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:01:27.000456 23087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:01:27.000470 23087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:01:27.006173 23087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46587,127.0.98.61:43565,127.0.98.60:39817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:01:27.007385 23087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:01:27.008400 23087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:01:27.017930 23095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:27.017930 23092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:27.018075 23093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:01:27.018368 23087 server_base.cc:1034] running on GCE node
I20260927 15:01:27.019016 23087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:01:27.020236 23087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:01:27.021435 23087 hybrid_clock.cc:648] HybridClock initialized: now 1790521287021414 us; error 37 us; skew 500 ppm
Sep 27 15:01:27 dist-test-slave-pz0h krb5kdc[22999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:01:27.024678 23087 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 15:01:27.025725 23087 webserver.cc:460] Webserver started at http://127.0.98.61:38279/ using document root <none> and password file <none>
I20260927 15:01:27.026300 23087 fs_manager.cc:359] Metadata directory not provided
I20260927 15:01:27.026372 23087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:01:27.026587 23087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:01:27.028926 23087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-1/data/instance:
uuid: "1747c3eaeabe480dbd556790e01afa1e"
format_stamp: "Formatted at 2026-09-27 15:01:27 on dist-test-slave-pz0h"
server_key: "5bdade6745ec5f3778cab59327d5ad73"
server_key_iv: "1aaf23125051557fca14b850e8fe6379"
server_key_version: "encryptionkey@0"
I20260927 15:01:27.029532 23087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-1/wal/instance:
uuid: "1747c3eaeabe480dbd556790e01afa1e"
format_stamp: "Formatted at 2026-09-27 15:01:27 on dist-test-slave-pz0h"
server_key: "5bdade6745ec5f3778cab59327d5ad73"
server_key_iv: "1aaf23125051557fca14b850e8fe6379"
server_key_version: "encryptionkey@0"
I20260927 15:01:27.033628 23087 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:01:27.036392 23102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:01:27.037724 23087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:01:27.037869 23087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-1/wal
uuid: "1747c3eaeabe480dbd556790e01afa1e"
format_stamp: "Formatted at 2026-09-27 15:01:27 on dist-test-slave-pz0h"
server_key: "5bdade6745ec5f3778cab59327d5ad73"
server_key_iv: "1aaf23125051557fca14b850e8fe6379"
server_key_version: "encryptionkey@0"
I20260927 15:01:27.037988 23087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:01:27.059801 23087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:01:27.063486 23087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:01:27.063704 23087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:01:27.072567 23087 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43565
I20260927 15:01:27.072882 23154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43565 every 8 connection(s)
I20260927 15:01:27.073794 23087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-1/data/info.pb
I20260927 15:01:27.077126 23155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:01:27.081740 23155 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46587 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:01:27.083673 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 23087
I20260927 15:01:27.083792 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-1/wal/instance
I20260927 15:01:27.084203 392 external_mini_cluster.cc:1428] Setting key 71f0f44d6fc6751d52e09fb90dff8759
I20260927 15:01:27.084563 23080 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43565 } attempt: 3
Sep 27 15:01:27 dist-test-slave-pz0h krb5kdc[22999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Sep 27 15:01:27 dist-test-slave-pz0h krb5kdc[22999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260927 15:01:27.097023 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.085014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43565 (local address 127.0.0.1:48370)
0927 15:01:27.085224 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:27.085241 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:01:27.085334 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:01:27.087416 (+ 2082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:01:27.087419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:01:27.087635 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:01:27.087763 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:27.087771 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.089406 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.089409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:27.089936 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:27.089943 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.090883 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.090887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:27.090995 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:27.091577 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:01:27.091592 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:01:27.093019 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:01:27.095026 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:27.095029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:27.095032 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:27.095285 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:27.095675 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:27.095677 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:27.095679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:27.095756 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:27.096469 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:01:27.096472 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:01:27.096590 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 15:01:27.096690 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 15:01:27.096833 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260927 15:01:27.097837 23084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.084181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46020 (local address 127.0.98.62:46587)
0927 15:01:27.084356 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:27.084361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:27.085826 (+ 1465us) server_negotiation.cc:407] Connection header received
0927 15:01:27.086996 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:27.087000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:27.087063 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:27.087137 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:27.088296 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.089061 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.090565 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.090968 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.091062 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:27.093867 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:27.093870 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:27.093872 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:27.093894 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:27.096026 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:27.096521 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:27.096524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:27.096526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:27.096603 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:27.096888 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:27.096892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:27.096893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:27.097106 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:27.097179 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:27.097591 (+ 412us) server_negotiation.cc:299] Negotiation successful
0927 15:01:27.097707 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:01:27.098284 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.085733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48370 (local address 127.0.98.61:43565)
0927 15:01:27.086045 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:27.086056 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:27.086103 (+ 47us) server_negotiation.cc:407] Connection header received
0927 15:01:27.086876 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:27.087016 (+ 140us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:27.087251 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:27.087438 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:27.087964 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.089284 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.090047 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.090710 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.090916 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:27.093251 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:27.093259 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:27.093272 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:27.093296 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:27.094923 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:27.095412 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:27.095418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:27.095425 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:27.095560 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:27.095835 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:27.095839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:27.095841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:27.096346 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:27.096492 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:27.096875 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 15:01:27.097212 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":133,"server-negotiator.queue_time_us":204,"spinlock_wait_cycles":3840,"thread_start_us":111,"threads_started":1}
I20260927 15:01:27.098395 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.084675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46587 (local address 127.0.0.1:46020)
0927 15:01:27.085673 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:27.085733 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 15:01:27.086743 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:01:27.087243 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:01:27.087253 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:01:27.087704 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:01:27.088148 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:27.088164 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.089190 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.089194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:27.090423 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:27.090435 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.091185 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.091187 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:27.091285 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:27.091750 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:01:27.091766 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:01:27.093624 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:01:27.096135 (+ 2511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:27.096142 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:27.096147 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:27.096399 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:27.096699 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:27.096702 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:27.096704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:27.096788 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:27.097254 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:01:27.097260 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:01:27.097371 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 15:01:27.098022 (+ 651us) client_negotiation.cc:240] Negotiation successful
0927 15:01:27.098213 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":26,"thread_start_us":164,"threads_started":1}
I20260927 15:01:27.100112 23155 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43565 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:01:27.100518 23080 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39817 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:01:27.101307 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.101071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39817 (local address 127.0.0.1:45024)
0927 15:01:27.101181 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:27.101238 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 15:01:27.102058 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.101782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39817 (local address 127.0.0.1:45032)
0927 15:01:27.101935 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:27.101987 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260927 15:01:27.102169 23080 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39817: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39817: connect: Connection refused (error 111)
Sep 27 15:01:27 dist-test-slave-pz0h krb5kdc[22999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 15:01:27.110473 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.100690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43565 (local address 127.0.0.1:48378)
0927 15:01:27.100933 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:27.100948 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:01:27.101057 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:01:27.101728 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:01:27.101731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:01:27.101925 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:01:27.102035 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:27.102043 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.103267 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.103270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:27.103811 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:27.103819 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.104406 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.104411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:27.104557 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:27.105240 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:01:27.105257 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:01:27.106623 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:01:27.108626 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:27.108630 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:27.108633 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:27.108825 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:27.109066 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:27.109069 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:27.109071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:27.109145 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:27.109509 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:01:27.109512 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:01:27.109740 (+ 228us) client_negotiation.cc:769] Sending connection context
0927 15:01:27.109919 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 15:01:27.110137 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260927 15:01:27.110473 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.100820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48378 (local address 127.0.98.61:43565)
0927 15:01:27.100936 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:27.100940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:27.100993 (+ 53us) server_negotiation.cc:407] Connection header received
0927 15:01:27.101588 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:27.101590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:27.101631 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:27.101703 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:27.102286 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.103125 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.103931 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.104182 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.104269 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:27.106840 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:27.106845 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:27.106848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:27.106876 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:27.108496 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:27.108911 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:27.108915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:27.108917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:27.108987 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:27.109220 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:27.109223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:27.109224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:27.109422 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:27.109509 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:27.110010 (+ 501us) server_negotiation.cc:299] Negotiation successful
0927 15:01:27.110223 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":42}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 15:01:27.111935 23155 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39817 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:01:27.112020 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDguobN/build/debug/bin/kudu
/tmp/dist-test-taskDguobN/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39817
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46587,127.0.98.61:43565,127.0.98.60:39817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:01:27.112862 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.112507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39817 (local address 127.0.0.1:45046)
0927 15:01:27.112690 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:27.112780 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260927 15:01:27.113296 23105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39817: connect: Connection refused (error 111)
I20260927 15:01:27.114434 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.114126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39817 (local address 127.0.0.1:45062)
0927 15:01:27.114280 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:27.114344 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 15:01:27.114615 23155 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39817: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39817: connect: Connection refused (error 111)
I20260927 15:01:27.133379 23080 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39817 } attempt: 1
I20260927 15:01:27.134348 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.134024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39817 (local address 127.0.0.1:45078)
0927 15:01:27.134184 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:27.134259 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 15:01:27.135282 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.135033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39817 (local address 127.0.0.1:45094)
0927 15:01:27.135154 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:27.135208 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 15:01:27.135440 23080 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39817: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39817: connect: Connection refused (error 111)
I20260927 15:01:27.163949 23155 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39817 } attempt: 1
I20260927 15:01:27.164880 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.164580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39817 (local address 127.0.0.1:45110)
0927 15:01:27.164737 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:27.164799 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 15:01:27.165573 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.165392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39817 (local address 127.0.0.1:45112)
0927 15:01:27.165476 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:27.165519 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 15:01:27.165714 23155 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39817: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39817: connect: Connection refused (error 111)
I20260927 15:01:27.210615 23080 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39817 } attempt: 2
I20260927 15:01:27.211546 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.211236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39817 (local address 127.0.0.1:45114)
0927 15:01:27.211399 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:27.211465 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 15:01:27.212232 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.212041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39817 (local address 127.0.0.1:45124)
0927 15:01:27.212133 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:27.212179 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 15:01:27.212390 23080 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39817: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39817: connect: Connection refused (error 111)
W20260927 15:01:27.229272 23161 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:01:27.229614 23161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:01:27.229660 23161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:01:27.229725 23161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 15:01:27.233896 23155 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39817 } attempt: 2
W20260927 15:01:27.234207 23161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:01:27.234265 23161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:01:27.234285 23161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:01:27.234316 23161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:01:27.234342 23161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:01:27.234776 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.234498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39817 (local address 127.0.0.1:45140)
0927 15:01:27.234634 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:27.234695 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 15:01:27.235433 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.235235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39817 (local address 127.0.0.1:45144)
0927 15:01:27.235327 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:27.235374 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 15:01:27.235586 23155 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39817: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39817: connect: Connection refused (error 111)
I20260927 15:01:27.240266 23161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46587,127.0.98.61:43565,127.0.98.60:39817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:15:17 UTC on ccycloud.quasar-ledxfe.root.comops.site
build id 4676134
I20260927 15:01:27.241423 23161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:01:27.242462 23161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:01:27.251123 23167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:27.251125 23170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:01:27.251125 23168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:01:27.251202 23161 server_base.cc:1034] running on GCE node
I20260927 15:01:27.252055 23161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:01:27.253340 23161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:01:27.254534 23161 hybrid_clock.cc:648] HybridClock initialized: now 1790521287254519 us; error 36 us; skew 500 ppm
Sep 27 15:01:27 dist-test-slave-pz0h krb5kdc[22999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:01:27.257542 23161 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 15:01:27.258564 23161 webserver.cc:460] Webserver started at http://127.0.98.60:40935/ using document root <none> and password file <none>
I20260927 15:01:27.259186 23161 fs_manager.cc:359] Metadata directory not provided
I20260927 15:01:27.259260 23161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:01:27.259470 23161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:01:27.261673 23161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-2/data/instance:
uuid: "38f7badfd80a4a488b8c4c3ab0b4dd46"
format_stamp: "Formatted at 2026-09-27 15:01:27 on dist-test-slave-pz0h"
server_key: "57e4c668b9639a8ab0f287f6c679a870"
server_key_iv: "387ee085f76f13f1613591bf4a762146"
server_key_version: "encryptionkey@0"
I20260927 15:01:27.262285 23161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-2/wal/instance:
uuid: "38f7badfd80a4a488b8c4c3ab0b4dd46"
format_stamp: "Formatted at 2026-09-27 15:01:27 on dist-test-slave-pz0h"
server_key: "57e4c668b9639a8ab0f287f6c679a870"
server_key_iv: "387ee085f76f13f1613591bf4a762146"
server_key_version: "encryptionkey@0"
I20260927 15:01:27.266203 23161 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:01:27.268826 23177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:01:27.270238 23161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:01:27.270388 23161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-2/wal
uuid: "38f7badfd80a4a488b8c4c3ab0b4dd46"
format_stamp: "Formatted at 2026-09-27 15:01:27 on dist-test-slave-pz0h"
server_key: "57e4c668b9639a8ab0f287f6c679a870"
server_key_iv: "387ee085f76f13f1613591bf4a762146"
server_key_version: "encryptionkey@0"
I20260927 15:01:27.270510 23161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:01:27.296792 23161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:01:27.300172 23161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:01:27.300370 23161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:01:27.307552 23161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39817
I20260927 15:01:27.307823 23229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39817 every 8 connection(s)
I20260927 15:01:27.308571 23161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-2/data/info.pb
I20260927 15:01:27.309862 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDguobN/build/debug/bin/kudu as pid 23161
I20260927 15:01:27.309958 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDguobN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790521004237705-392-0/minicluster-data/master-2/wal/instance
I20260927 15:01:27.310245 392 external_mini_cluster.cc:1428] Setting key 7dceec429349b0a09ad8addcec53825a
I20260927 15:01:27.311539 23080 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39817 } attempt: 3
I20260927 15:01:27.311790 23230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:01:27.316776 23230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46587 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:01:27 dist-test-slave-pz0h krb5kdc[22999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 15:01:27 dist-test-slave-pz0h krb5kdc[22999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 15:01:27.325122 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.312013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39817 (local address 127.0.0.1:45162)
0927 15:01:27.312136 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:27.312151 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:01:27.312235 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:01:27.313697 (+ 1462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:01:27.313700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:01:27.313909 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:01:27.314029 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:27.314038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.315768 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.315773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:27.316988 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:27.317012 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.317992 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.317997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:27.318105 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:27.318882 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:01:27.318897 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:01:27.320410 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:01:27.323556 (+ 3146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:27.323559 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:27.323560 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:27.323762 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:27.324055 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:27.324058 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:27.324060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:27.324154 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:27.324614 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:01:27.324618 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:01:27.324719 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 15:01:27.324802 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 15:01:27.324939 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
Sep 27 15:01:27 dist-test-slave-pz0h krb5kdc[22999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:01:27.326462 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.312356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45162 (local address 127.0.98.60:39817)
0927 15:01:27.312655 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:27.312667 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:27.312680 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:01:27.312977 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:27.312993 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:27.313390 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:27.313730 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:27.314265 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.315638 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.317160 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.317835 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.318342 (+ 507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:27.320591 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:27.320599 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:27.320610 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:27.320643 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:27.323105 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:27.323849 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:27.323851 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:27.323852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:27.323933 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:27.324267 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:27.324269 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:27.324271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:27.324495 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:27.325441 (+ 946us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:27.325512 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 15:01:27.325642 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":227,"spinlock_wait_cycles":18176,"thread_start_us":65,"threads_started":1}
I20260927 15:01:27.326467 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.311578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45154 (local address 127.0.98.60:39817)
0927 15:01:27.312021 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:27.312041 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:27.312091 (+ 50us) server_negotiation.cc:407] Connection header received
0927 15:01:27.312968 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:27.312992 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:27.313392 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:27.314181 (+ 789us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:27.314598 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.315661 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.317253 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.317828 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.317984 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:27.321204 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:27.321208 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:27.321211 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:27.321239 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:27.323090 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:27.323528 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:27.323533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:27.323551 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:27.323667 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:27.323940 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:27.323943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:27.323944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:27.324495 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:27.324707 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:27.325051 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 15:01:27.325374 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":270,"server-negotiator.queue_time_us":296,"spinlock_wait_cycles":7808,"thread_start_us":146,"threads_started":1}
I20260927 15:01:27.326743 23155 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39817 } attempt: 3
I20260927 15:01:27.330159 23084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.318915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46028 (local address 127.0.98.62:46587)
0927 15:01:27.319054 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:27.319058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:27.319490 (+ 432us) server_negotiation.cc:407] Connection header received
0927 15:01:27.319727 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:27.319731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:27.319790 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:27.319882 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:27.320582 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.321283 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.322388 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.322626 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.322715 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:27.326069 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:27.326073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:27.326076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:27.326101 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:27.328156 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:27.328693 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:27.328697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:27.328699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:27.328786 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:27.329089 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:27.329092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:27.329094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:27.329369 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:27.329444 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:27.329874 (+ 430us) server_negotiation.cc:299] Negotiation successful
0927 15:01:27.330001 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:01:27.330180 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.318855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46587 (local address 127.0.0.1:46028)
0927 15:01:27.319347 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:27.319411 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 15:01:27.319601 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:01:27.319939 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:01:27.319944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:01:27.320349 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:01:27.320479 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:27.320487 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.321388 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.321392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:27.322273 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:27.322285 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.322733 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.322737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:27.322878 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:27.323947 (+ 1069us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:01:27.323975 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:01:27.325936 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:01:27.328259 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:27.328265 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:27.328270 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:27.328573 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:27.328898 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:27.328900 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:27.328902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:27.328994 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:27.329526 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:01:27.329532 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:01:27.329670 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 15:01:27.329811 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 15:01:27.329999 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"spinlock_wait_cycles":1792,"thread_start_us":107,"threads_started":1}
I20260927 15:01:27.331359 23230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43565 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:01:27.331606 23080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adc5a1b8348b4bbcaa43e1df788317c8: Bootstrap starting.
Sep 27 15:01:27 dist-test-slave-pz0h krb5kdc[22999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 15:01:27.334558 23080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adc5a1b8348b4bbcaa43e1df788317c8: Neither blocks nor log segments found. Creating new log.
I20260927 15:01:27.336351 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.327467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45164 (local address 127.0.98.60:39817)
0927 15:01:27.327547 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:27.327551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:27.327562 (+ 11us) server_negotiation.cc:407] Connection header received
0927 15:01:27.327597 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:27.327599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:27.327643 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:27.327725 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:27.328976 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.329656 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.330655 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.330992 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.331096 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:27.333206 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:27.333209 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:27.333212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:27.333236 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:27.334747 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:27.335180 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:27.335184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:27.335186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:27.335262 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:27.335502 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:27.335506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:27.335508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:27.335738 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:27.335831 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:27.336096 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 15:01:27.336208 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 15:01:27.336724 23080 log.cc:824] T 00000000000000000000000000000000 P adc5a1b8348b4bbcaa43e1df788317c8: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:01:27 dist-test-slave-pz0h krb5kdc[22999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 15:01:27.337200 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.327215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39817 (local address 127.0.0.1:45164)
0927 15:01:27.327364 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:27.327380 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:01:27.327460 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:01:27.328570 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:01:27.328573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:01:27.328755 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:01:27.328877 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:27.328886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.329763 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.329766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:27.330532 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:27.330543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.331117 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.331120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:27.331206 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:27.331711 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:01:27.331722 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:01:27.333075 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:01:27.334878 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:27.334882 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:27.334884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:27.335093 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:27.335343 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:27.335346 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:27.335348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:27.335424 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:27.335836 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:01:27.335839 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:01:27.335936 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 15:01:27.336029 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 15:01:27.336178 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"spinlock_wait_cycles":768}
I20260927 15:01:27.340013 23080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adc5a1b8348b4bbcaa43e1df788317c8: No bootstrap required, opened a new log
I20260927 15:01:27.341092 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.331988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48388 (local address 127.0.98.61:43565)
0927 15:01:27.332095 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:27.332100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:27.332224 (+ 124us) server_negotiation.cc:407] Connection header received
0927 15:01:27.332311 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:27.332314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:27.332362 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:27.332439 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:27.332866 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.333482 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.334606 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.334951 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.335031 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:27.337505 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:27.337510 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:27.337512 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:27.337537 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:27.339134 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:27.339653 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:27.339657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:27.339658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:27.339763 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:27.340076 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:27.340079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:27.340081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:27.340350 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:27.340423 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:27.340803 (+ 380us) server_negotiation.cc:299] Negotiation successful
0927 15:01:27.340930 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 15:01:27 dist-test-slave-pz0h krb5kdc[22999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 15:01:27.341557 23155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1747c3eaeabe480dbd556790e01afa1e: Bootstrap starting.
I20260927 15:01:27.341606 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.331873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43565 (local address 127.0.0.1:48388)
0927 15:01:27.332141 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:27.332155 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:01:27.332239 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:01:27.332465 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:01:27.332468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:01:27.332649 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:01:27.332763 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:27.332770 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.333678 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.333682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:27.334447 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:27.334459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.335177 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.335180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:27.335280 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:27.335893 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:01:27.335905 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:01:27.337357 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:01:27.339266 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:27.339270 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:27.339272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:27.339522 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:27.339879 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:27.339882 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:27.339884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:27.339973 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:27.340500 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:01:27.340503 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:01:27.340622 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 15:01:27.341280 (+ 658us) client_negotiation.cc:240] Negotiation successful
0927 15:01:27.341442 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260927 15:01:27.344671 23155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1747c3eaeabe480dbd556790e01afa1e: Neither blocks nor log segments found. Creating new log.
I20260927 15:01:27.345037 23080 raft_consensus.cc:348] T 00000000000000000000000000000000 P adc5a1b8348b4bbcaa43e1df788317c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc5a1b8348b4bbcaa43e1df788317c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46587 } } peers { permanent_uuid: "1747c3eaeabe480dbd556790e01afa1e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43565 } } peers { permanent_uuid: "38f7badfd80a4a488b8c4c3ab0b4dd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39817 } }
I20260927 15:01:27.345495 23080 raft_consensus.cc:729] T 00000000000000000000000000000000 P adc5a1b8348b4bbcaa43e1df788317c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc5a1b8348b4bbcaa43e1df788317c8, State: Initialized, Role: FOLLOWER
I20260927 15:01:27.345767 23155 log.cc:824] T 00000000000000000000000000000000 P 1747c3eaeabe480dbd556790e01afa1e: Log is configured to *not* fsync() on all Append() calls
I20260927 15:01:27.345891 23080 consensus_queue.cc:260] T 00000000000000000000000000000000 P adc5a1b8348b4bbcaa43e1df788317c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc5a1b8348b4bbcaa43e1df788317c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46587 } } peers { permanent_uuid: "1747c3eaeabe480dbd556790e01afa1e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43565 } } peers { permanent_uuid: "38f7badfd80a4a488b8c4c3ab0b4dd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39817 } }
I20260927 15:01:27.346339 23084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.329340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46044 (local address 127.0.98.62:46587)
0927 15:01:27.330710 (+ 1370us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:27.330714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:27.335647 (+ 4933us) server_negotiation.cc:407] Connection header received
0927 15:01:27.335701 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:27.335703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:27.335749 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:27.335820 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:27.337082 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.337759 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.338942 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.339242 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.339342 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:27.342208 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:27.342212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:27.342215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:27.342245 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:27.343920 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:27.344448 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:27.344452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:27.344454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:27.344548 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:27.344893 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:27.344896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:27.344898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:27.345146 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:27.345334 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:27.346042 (+ 708us) server_negotiation.cc:299] Negotiation successful
0927 15:01:27.346169 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1312,"thread_start_us":93,"threads_started":1}
I20260927 15:01:27.347378 23238 sys_catalog.cc:455] T 00000000000000000000000000000000 P adc5a1b8348b4bbcaa43e1df788317c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc5a1b8348b4bbcaa43e1df788317c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46587 } } peers { permanent_uuid: "1747c3eaeabe480dbd556790e01afa1e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43565 } } peers { permanent_uuid: "38f7badfd80a4a488b8c4c3ab0b4dd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39817 } } }
I20260927 15:01:27.347513 23080 sys_catalog.cc:564] T 00000000000000000000000000000000 P adc5a1b8348b4bbcaa43e1df788317c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:01:27.347637 23238 sys_catalog.cc:458] T 00000000000000000000000000000000 P adc5a1b8348b4bbcaa43e1df788317c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:01:27.348131 23230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39817 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:01:27.349017 23155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1747c3eaeabe480dbd556790e01afa1e: No bootstrap required, opened a new log
I20260927 15:01:27.353595 23155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1747c3eaeabe480dbd556790e01afa1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc5a1b8348b4bbcaa43e1df788317c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46587 } } peers { permanent_uuid: "1747c3eaeabe480dbd556790e01afa1e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43565 } } peers { permanent_uuid: "38f7badfd80a4a488b8c4c3ab0b4dd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39817 } }
I20260927 15:01:27.353935 23155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1747c3eaeabe480dbd556790e01afa1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1747c3eaeabe480dbd556790e01afa1e, State: Initialized, Role: FOLLOWER
I20260927 15:01:27.354398 23155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1747c3eaeabe480dbd556790e01afa1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc5a1b8348b4bbcaa43e1df788317c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46587 } } peers { permanent_uuid: "1747c3eaeabe480dbd556790e01afa1e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43565 } } peers { permanent_uuid: "38f7badfd80a4a488b8c4c3ab0b4dd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39817 } }
Sep 27 15:01:27 dist-test-slave-pz0h krb5kdc[22999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 15:01:27.355087 23251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1747c3eaeabe480dbd556790e01afa1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc5a1b8348b4bbcaa43e1df788317c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46587 } } peers { permanent_uuid: "1747c3eaeabe480dbd556790e01afa1e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43565 } } peers { permanent_uuid: "38f7badfd80a4a488b8c4c3ab0b4dd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39817 } } }
I20260927 15:01:27.355237 23251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1747c3eaeabe480dbd556790e01afa1e [sys.catalog]: This master's current role is: FOLLOWER
Sep 27 15:01:27 dist-test-slave-pz0h krb5kdc[22999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790521286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 15:01:27.356189 23155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1747c3eaeabe480dbd556790e01afa1e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:01:27.354004 23012 master_runner.cc:186] Error getting master registration for 127.0.98.61:43565: OK, instance_id { permanent_uuid: "1747c3eaeabe480dbd556790e01afa1e" instance_seqno: 1790521287069550 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:01:27.358466 23012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:01:27.360091 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.349141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39817 (local address 127.0.0.1:45178)
0927 15:01:27.349293 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 15:01:27.349308 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:01:27.349396 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:01:27.349737 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:01:27.349741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:01:27.349948 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:01:27.350058 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:27.350066 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.351144 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.351147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:27.351888 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:01:27.351899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.352417 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.352421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:01:27.352497 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:27.353281 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:01:27.353297 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:01:27.355824 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:01:27.358514 (+ 2690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:27.358518 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:27.358520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:27.358788 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:27.359190 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:01:27.359193 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:01:27.359195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:01:27.359267 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:01:27.359639 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:01:27.359642 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:01:27.359722 (+ 80us) client_negotiation.cc:769] Sending connection context
0927 15:01:27.359798 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 15:01:27.359930 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 15:01:27.360332 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.347919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48394 (local address 127.0.98.61:43565)
0927 15:01:27.348087 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:27.348099 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:27.348903 (+ 804us) server_negotiation.cc:407] Connection header received
0927 15:01:27.349056 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:27.349060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:27.349115 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:27.349206 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:27.350269 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.351058 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.352184 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.352531 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.352609 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:27.356054 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:27.356059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:27.356062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:27.356091 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:27.357951 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:27.358549 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:27.358553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:27.358555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:27.358644 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:27.358975 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:27.358978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:27.358980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:27.359193 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:27.359363 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:27.360074 (+ 711us) server_negotiation.cc:299] Negotiation successful
0927 15:01:27.360194 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 15:01:27.361053 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.349403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45178 (local address 127.0.98.60:39817)
0927 15:01:27.349518 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:27.349522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:27.349540 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:01:27.349576 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:27.349579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:27.349633 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:27.350164 (+ 531us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:27.350249 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.351033 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.352016 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.352305 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.355989 (+ 3684us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:27.356122 (+ 133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:27.356126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:27.356129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:27.356148 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:27.358259 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:27.359017 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:27.359020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:27.359022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:27.359104 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:27.359346 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:27.359349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:27.359350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:27.359539 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:27.360704 (+ 1165us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:27.360796 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 15:01:27.360916 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 15:01:27.362931 23250 catalog_manager.cc:1558] T 00000000000000000000000000000000 P adc5a1b8348b4bbcaa43e1df788317c8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:01:27.363015 23250 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:01:27.363785 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:01:27.364826 23230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f7badfd80a4a488b8c4c3ab0b4dd46: Bootstrap starting.
I20260927 15:01:27.364902 23054 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "adc5a1b8348b4bbcaa43e1df788317c8"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46044
I20260927 15:01:27.365125 23054 raft_consensus.cc:482] T 00000000000000000000000000000000 P adc5a1b8348b4bbcaa43e1df788317c8 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 15:01:27.365238 23054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adc5a1b8348b4bbcaa43e1df788317c8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:01:27.366654 23054 raft_consensus.cc:504] T 00000000000000000000000000000000 P adc5a1b8348b4bbcaa43e1df788317c8 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc5a1b8348b4bbcaa43e1df788317c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46587 } } peers { permanent_uuid: "1747c3eaeabe480dbd556790e01afa1e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43565 } } peers { permanent_uuid: "38f7badfd80a4a488b8c4c3ab0b4dd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39817 } }
I20260927 15:01:27.368273 23230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38f7badfd80a4a488b8c4c3ab0b4dd46: Neither blocks nor log segments found. Creating new log.
I20260927 15:01:27.366003 23087 master_runner.cc:186] Error getting master registration for 127.0.98.60:39817: OK, instance_id { permanent_uuid: "38f7badfd80a4a488b8c4c3ab0b4dd46" instance_seqno: 1790521287305480 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:01:27.369158 23087 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:01:27.369105 23129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "adc5a1b8348b4bbcaa43e1df788317c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1747c3eaeabe480dbd556790e01afa1e"
I20260927 15:01:27.369416 23129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1747c3eaeabe480dbd556790e01afa1e [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:01:27.369663 23230 log.cc:824] T 00000000000000000000000000000000 P 38f7badfd80a4a488b8c4c3ab0b4dd46: Log is configured to *not* fsync() on all Append() calls
I20260927 15:01:27.370819 23129 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1747c3eaeabe480dbd556790e01afa1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adc5a1b8348b4bbcaa43e1df788317c8 in term 1.
I20260927 15:01:27.371434 23028 leader_election.cc:302] T 00000000000000000000000000000000 P adc5a1b8348b4bbcaa43e1df788317c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1747c3eaeabe480dbd556790e01afa1e, adc5a1b8348b4bbcaa43e1df788317c8; no voters:
I20260927 15:01:27.371757 23238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adc5a1b8348b4bbcaa43e1df788317c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:01:27.371981 23238 raft_consensus.cc:686] T 00000000000000000000000000000000 P adc5a1b8348b4bbcaa43e1df788317c8 [term 1 LEADER]: Becoming Leader. State: Replica: adc5a1b8348b4bbcaa43e1df788317c8, State: Running, Role: LEADER
I20260927 15:01:27.372289 23054 leader_election.cc:288] T 00000000000000000000000000000000 P adc5a1b8348b4bbcaa43e1df788317c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1747c3eaeabe480dbd556790e01afa1e (127.0.98.61:43565), 38f7badfd80a4a488b8c4c3ab0b4dd46 (127.0.98.60:39817)
I20260927 15:01:27.372349 23238 consensus_queue.cc:237] T 00000000000000000000000000000000 P adc5a1b8348b4bbcaa43e1df788317c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc5a1b8348b4bbcaa43e1df788317c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46587 } } peers { permanent_uuid: "1747c3eaeabe480dbd556790e01afa1e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43565 } } peers { permanent_uuid: "38f7badfd80a4a488b8c4c3ab0b4dd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39817 } }
I20260927 15:01:27.372233 23230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f7badfd80a4a488b8c4c3ab0b4dd46: No bootstrap required, opened a new log
W20260927 15:01:27.373812 23262 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1747c3eaeabe480dbd556790e01afa1e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:01:27.373898 23262 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:01:27.374483 23238 sys_catalog.cc:455] T 00000000000000000000000000000000 P adc5a1b8348b4bbcaa43e1df788317c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adc5a1b8348b4bbcaa43e1df788317c8. Latest consensus state: current_term: 1 leader_uuid: "adc5a1b8348b4bbcaa43e1df788317c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc5a1b8348b4bbcaa43e1df788317c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46587 } } peers { permanent_uuid: "1747c3eaeabe480dbd556790e01afa1e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43565 } } peers { permanent_uuid: "38f7badfd80a4a488b8c4c3ab0b4dd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39817 } } }
I20260927 15:01:27.374617 23238 sys_catalog.cc:458] T 00000000000000000000000000000000 P adc5a1b8348b4bbcaa43e1df788317c8 [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:27.377825 23230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38f7badfd80a4a488b8c4c3ab0b4dd46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc5a1b8348b4bbcaa43e1df788317c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46587 } } peers { permanent_uuid: "1747c3eaeabe480dbd556790e01afa1e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43565 } } peers { permanent_uuid: "38f7badfd80a4a488b8c4c3ab0b4dd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39817 } }
I20260927 15:01:27.378175 23230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38f7badfd80a4a488b8c4c3ab0b4dd46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38f7badfd80a4a488b8c4c3ab0b4dd46, State: Initialized, Role: FOLLOWER
I20260927 15:01:27.378580 23230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38f7badfd80a4a488b8c4c3ab0b4dd46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc5a1b8348b4bbcaa43e1df788317c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46587 } } peers { permanent_uuid: "1747c3eaeabe480dbd556790e01afa1e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43565 } } peers { permanent_uuid: "38f7badfd80a4a488b8c4c3ab0b4dd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39817 } }
I20260927 15:01:27.379443 23273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f7badfd80a4a488b8c4c3ab0b4dd46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc5a1b8348b4bbcaa43e1df788317c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46587 } } peers { permanent_uuid: "1747c3eaeabe480dbd556790e01afa1e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43565 } } peers { permanent_uuid: "38f7badfd80a4a488b8c4c3ab0b4dd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39817 } } }
I20260927 15:01:27.379806 23273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f7badfd80a4a488b8c4c3ab0b4dd46 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:01:27.380609 23230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38f7badfd80a4a488b8c4c3ab0b4dd46 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:01:27.380676 23204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "adc5a1b8348b4bbcaa43e1df788317c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "38f7badfd80a4a488b8c4c3ab0b4dd46"
I20260927 15:01:27.380925 23204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38f7badfd80a4a488b8c4c3ab0b4dd46 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:01:27.382947 23204 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 38f7badfd80a4a488b8c4c3ab0b4dd46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adc5a1b8348b4bbcaa43e1df788317c8 in term 1.
I20260927 15:01:27.384673 23272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:01:27.385737 23272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:01:27.395372 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.383223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48396 (local address 127.0.98.61:43565)
0927 15:01:27.383357 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:27.383362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:27.386273 (+ 2911us) server_negotiation.cc:407] Connection header received
0927 15:01:27.386383 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:27.386387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:27.386445 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:27.386524 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:27.387548 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.388241 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.389348 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.389651 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.389727 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:27.391663 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:27.391668 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:27.391670 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:27.391699 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:27.393393 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:27.393914 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:27.393918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:27.393920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:27.394012 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:27.394304 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:27.394307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:27.394309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:27.394540 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:27.394677 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:27.395059 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 15:01:27.395195 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:01:27.402098 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.384632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45194 (local address 127.0.98.60:39817)
0927 15:01:27.384761 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:27.384766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:27.386560 (+ 1794us) server_negotiation.cc:407] Connection header received
0927 15:01:27.386716 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:27.386719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:27.386768 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:27.386852 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:27.390747 (+ 3895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.391522 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.395508 (+ 3986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.395872 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.395959 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:27.398044 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:27.398056 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:27.398059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:27.398091 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:27.399736 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:27.400251 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:27.400255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:27.400257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:27.400347 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:27.400930 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:27.400938 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:27.400940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:27.401214 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:27.401352 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:27.401770 (+ 418us) server_negotiation.cc:299] Negotiation successful
0927 15:01:27.401909 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:01:27.404222 23084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:27.385415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46058 (local address 127.0.98.62:46587)
0927 15:01:27.385545 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:27.385549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:27.387688 (+ 2139us) server_negotiation.cc:407] Connection header received
0927 15:01:27.388532 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:27.388536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:27.388597 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:27.388671 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:27.393088 (+ 4417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.393864 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.396695 (+ 2831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:27.397078 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:27.397163 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:27.399448 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:01:27.399452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:01:27.399456 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:01:27.399491 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:01:27.401194 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:27.401726 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:27.401731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:27.401733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:27.401828 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:01:27.402114 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:01:27.402118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:01:27.402120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:01:27.402346 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:01:27.402461 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:27.403881 (+ 1420us) server_negotiation.cc:299] Negotiation successful
0927 15:01:27.404037 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 15:01:27.408680 23204 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 38f7badfd80a4a488b8c4c3ab0b4dd46 [term 1 FOLLOWER]: Refusing update from remote peer adc5a1b8348b4bbcaa43e1df788317c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:01:27.408680 23129 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1747c3eaeabe480dbd556790e01afa1e [term 1 FOLLOWER]: Refusing update from remote peer adc5a1b8348b4bbcaa43e1df788317c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:01:27.410895 23238 consensus_queue.cc:1035] T 00000000000000000000000000000000 P adc5a1b8348b4bbcaa43e1df788317c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1747c3eaeabe480dbd556790e01afa1e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:01:27.411744 23238 consensus_queue.cc:1035] T 00000000000000000000000000000000 P adc5a1b8348b4bbcaa43e1df788317c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38f7badfd80a4a488b8c4c3ab0b4dd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:01:27.416464 23273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f7badfd80a4a488b8c4c3ab0b4dd46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adc5a1b8348b4bbcaa43e1df788317c8. Latest consensus state: current_term: 1 leader_uuid: "adc5a1b8348b4bbcaa43e1df788317c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc5a1b8348b4bbcaa43e1df788317c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46587 } } peers { permanent_uuid: "1747c3eaeabe480dbd556790e01afa1e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43565 } } peers { permanent_uuid: "38f7badfd80a4a488b8c4c3ab0b4dd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39817 } } }
I20260927 15:01:27.416644 23273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f7badfd80a4a488b8c4c3ab0b4dd46 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:01:27.417747 23251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1747c3eaeabe480dbd556790e01afa1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader adc5a1b8348b4bbcaa43e1df788317c8. Latest consensus state: current_term: 1 leader_uuid: "adc5a1b8348b4bbcaa43e1df788317c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc5a1b8348b4bbcaa43e1df788317c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46587 } } peers { permanent_uuid: "1747c3eaeabe480dbd556790e01afa1e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43565 } } peers { permanent_uuid: "38f7badfd80a4a488b8c4c3ab0b4dd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39817 } } }
I20260927 15:01:27.417932 23251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1747c3eaeabe480dbd556790e01afa1e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:01:27.421072 23238 sys_catalog.cc:455] T 00000000000000000000000000000000 P adc5a1b8348b4bbcaa43e1df788317c8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "adc5a1b8348b4bbcaa43e1df788317c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc5a1b8348b4bbcaa43e1df788317c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46587 } } peers { permanent_uuid: "1747c3eaeabe480dbd556790e01afa1e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43565 } } peers { permanent_uuid: "38f7badfd80a4a488b8c4c3ab0b4dd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39817 } } }
I20260927 15:01:27.421250 23238 sys_catalog.cc:458] T 00000000000000000000000000000000 P adc5a1b8348b4bbcaa43e1df788317c8 [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:27.421458 23288 mvcc.cc:205] Tried to move back new op lower bound from 7333975193217638400 to 7333975193081364480. Current Snapshot: MvccSnapshot[applied={T|T < 7333975193217638400}]
I20260927 15:01:27.421590 23251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1747c3eaeabe480dbd556790e01afa1e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "adc5a1b8348b4bbcaa43e1df788317c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc5a1b8348b4bbcaa43e1df788317c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46587 } } peers { permanent_uuid: "1747c3eaeabe480dbd556790e01afa1e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43565 } } peers { permanent_uuid: "38f7badfd80a4a488b8c4c3ab0b4dd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39817 } } }
I20260927 15:01:27.421736 23251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1747c3eaeabe480dbd556790e01afa1e [sys.catalog]: This master's current role is: FOLLOWER
W20260927 15:01:27.425414 23284 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 38f7badfd80a4a488b8c4c3ab0b4dd46: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:01:27.425544 23284 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:01:27.428081 23290 mvcc.cc:205] Tried to move back new op lower bound from 7333975193217638400 to 7333975193081364480. Current Snapshot: MvccSnapshot[applied={T|T < 7333975193217638400}]
I20260927 15:01:27.438540 23272 catalog_manager.cc:1347] Generated new cluster ID: 6b41bc5c45d2498990eeb935bf126f9c
I20260927 15:01:27.438666 23272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:01:27.439657 23238 sys_catalog.cc:455] T 00000000000000000000000000000000 P adc5a1b8348b4bbcaa43e1df788317c8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "adc5a1b8348b4bbcaa43e1df788317c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc5a1b8348b4bbcaa43e1df788317c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46587 } } peers { permanent_uuid: "1747c3eaeabe480dbd556790e01afa1e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43565 } } peers { permanent_uuid: "38f7badfd80a4a488b8c4c3ab0b4dd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39817 } } }
I20260927 15:01:27.439824 23238 sys_catalog.cc:458] T 00000000000000000000000000000000 P adc5a1b8348b4bbcaa43e1df788317c8 [sys.catalog]: This master's current role is: LEADER
I20260927 15:01:27.441404 23273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f7badfd80a4a488b8c4c3ab0b4dd46 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "adc5a1b8348b4bbcaa43e1df788317c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc5a1b8348b4bbcaa43e1df788317c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46587 } } peers { permanent_uuid: "1747c3eaeabe480dbd556790e01afa1e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43565 } } peers { permanent_uuid: "38f7badfd80a4a488b8c4c3ab0b4dd46" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39817 } } }
I20260927 15:01:27.441566 23273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f7badfd80a4a488b8c4c3ab0b4dd46 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:01:27.459802 23272 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:01:27.461249 23272 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:01:27.495362 23272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adc5a1b8348b4bbcaa43e1df788317c8: Generated new TSK 0
I20260927 15:01:27.496387 23272 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:01:27.536000 23084 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:01:27.532926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46074 (local address 127.0.98.62:46587)
0927 15:01:27.533098 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:27.533104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:27.534664 (+ 1560us) server_negotiation.cc:407] Connection header received
0927 15:01:27.534937 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:27.534942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:27.535019 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:27.535076 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:27.535817 (+ 741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46074: BlockingRecv error: recv got EOF from 127.0.0.1:46074 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260927 15:01:27.538367 23234 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:01:27.533935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45202 (local address 127.0.98.60:39817)
0927 15:01:27.534073 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:27.534078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:27.536505 (+ 2427us) server_negotiation.cc:407] Connection header received
0927 15:01:27.536950 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:27.536953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:27.537032 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:27.537090 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:27.538148 (+ 1058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45202: BlockingRecv error: recv got EOF from 127.0.0.1:45202 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 15:01:27.540907 23159 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:01:27.539276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48400 (local address 127.0.98.61:43565)
0927 15:01:27.539543 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:27.539549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:27.539918 (+ 369us) server_negotiation.cc:407] Connection header received
0927 15:01:27.540061 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:27.540065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:27.540122 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:27.540197 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:27.540744 (+ 547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48400: BlockingRecv error: recv got EOF from 127.0.0.1:48400 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 15:01:27.546327 23159 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:01:27.544498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48410 (local address 127.0.98.61:43565)
0927 15:01:27.544636 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:27.544640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:27.545183 (+ 543us) server_negotiation.cc:407] Connection header received
0927 15:01:27.545442 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:27.545446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:27.545544 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:27.545601 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:27.546196 (+ 595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48410: BlockingRecv error: recv got EOF from 127.0.0.1:48410 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 15:01:27.554656 23159 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:01:27.552787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48426 (local address 127.0.98.61:43565)
0927 15:01:27.552925 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:27.552929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:27.553595 (+ 666us) server_negotiation.cc:407] Connection header received
0927 15:01:27.553814 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:27.553818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:27.553872 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:27.553923 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:27.554525 (+ 602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48426: BlockingRecv error: recv got EOF from 127.0.0.1:48426 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 15:01:27.562706 23159 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:01:27.560669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48436 (local address 127.0.98.61:43565)
0927 15:01:27.560802 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:27.560806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:27.561420 (+ 614us) server_negotiation.cc:407] Connection header received
0927 15:01:27.561812 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:27.561816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:27.561878 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:27.561933 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:27.562578 (+ 645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48436: BlockingRecv error: recv got EOF from 127.0.0.1:48436 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 15:01:27.571808 23159 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:01:27.569797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48438 (local address 127.0.98.61:43565)
0927 15:01:27.569973 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:27.569978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:27.570623 (+ 645us) server_negotiation.cc:407] Connection header received
0927 15:01:27.570917 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:27.570921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:27.570981 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:27.571037 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:27.571668 (+ 631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48438: BlockingRecv error: recv got EOF from 127.0.0.1:48438 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260927 15:01:27.585093 23159 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:01:27.583201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48440 (local address 127.0.98.61:43565)
0927 15:01:27.583336 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:27.583340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:27.583977 (+ 637us) server_negotiation.cc:407] Connection header received
0927 15:01:27.584219 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:27.584223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:27.584283 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:27.584339 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:27.584964 (+ 625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48440: BlockingRecv error: recv got EOF from 127.0.0.1:48440 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 15:01:27.606246 23159 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:01:27.604531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48456 (local address 127.0.98.61:43565)
0927 15:01:27.604668 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:27.604672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:27.605285 (+ 613us) server_negotiation.cc:407] Connection header received
0927 15:01:27.605426 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:27.605429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:27.605494 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:27.605591 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:27.606117 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48456: BlockingRecv error: recv got EOF from 127.0.0.1:48456 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 15:01:27.641924 23159 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:01:27.640813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48468 (local address 127.0.98.61:43565)
0927 15:01:27.640988 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:27.640993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:27.641093 (+ 100us) server_negotiation.cc:407] Connection header received
0927 15:01:27.641246 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:27.641249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:27.641296 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:27.641367 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:27.641801 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48468: BlockingRecv error: recv got EOF from 127.0.0.1:48468 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 15:01:27.709502 23159 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:01:27.708246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48472 (local address 127.0.98.61:43565)
0927 15:01:27.708435 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:27.708440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:27.708614 (+ 174us) server_negotiation.cc:407] Connection header received
0927 15:01:27.708756 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:27.708759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:27.708813 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:27.708885 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:27.709383 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48472: BlockingRecv error: recv got EOF from 127.0.0.1:48472 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260927 15:01:27.840684 23159 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:01:27.839735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48484 (local address 127.0.98.61:43565)
0927 15:01:27.839851 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:27.839854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:27.839961 (+ 107us) server_negotiation.cc:407] Connection header received
0927 15:01:27.840079 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:27.840082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:27.840127 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:27.840207 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:27.840584 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48484: BlockingRecv error: recv got EOF from 127.0.0.1:48484 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 15:01:28.099920 23159 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:01:28.098887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48486 (local address 127.0.98.61:43565)
0927 15:01:28.099040 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:28.099045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:28.099062 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:01:28.099225 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:28.099228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:28.099278 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:28.099347 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:01:28.099796 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48486: BlockingRecv error: recv got EOF from 127.0.0.1:48486 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:01:28.375388 23262 catalog_manager.cc:1259] Loaded cluster ID: 6b41bc5c45d2498990eeb935bf126f9c
I20260927 15:01:28.375492 23262 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1747c3eaeabe480dbd556790e01afa1e: loading cluster ID for follower catalog manager: success
I20260927 15:01:28.377219 23262 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1747c3eaeabe480dbd556790e01afa1e: acquiring CA information for follower catalog manager: success
I20260927 15:01:28.378455 23262 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1747c3eaeabe480dbd556790e01afa1e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:01:28.426929 23284 catalog_manager.cc:1259] Loaded cluster ID: 6b41bc5c45d2498990eeb935bf126f9c
I20260927 15:01:28.427026 23284 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 38f7badfd80a4a488b8c4c3ab0b4dd46: loading cluster ID for follower catalog manager: success
I20260927 15:01:28.428685 23284 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 38f7badfd80a4a488b8c4c3ab0b4dd46: acquiring CA information for follower catalog manager: success
I20260927 15:01:28.429924 23284 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 38f7badfd80a4a488b8c4c3ab0b4dd46: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:01:28.618091 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:28.614001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48492 (local address 127.0.98.61:43565)
0927 15:01:28.614289 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:28.614293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:28.614308 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:01:28.614420 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:28.614423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:28.614550 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:28.614661 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:01:28.614984 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:28.615565 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:28.616493 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:28.616726 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:28.616806 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:28.617181 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:01:28.617674 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:01:28.617786 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:28.617867 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 15:01:28.617931 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":71,"threads_started":1}
I20260927 15:01:28.624634 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:28.621121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45206 (local address 127.0.98.60:39817)
0927 15:01:28.621374 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:28.621377 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:28.621392 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:01:28.621474 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:28.621476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:28.621598 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:28.621700 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:01:28.621861 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:28.622381 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:28.623231 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:28.623472 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:28.623559 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:28.623834 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:01:28.624289 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:01:28.624380 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:28.624409 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 15:01:28.624466 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":74,"threads_started":1}
I20260927 15:01:28.631994 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:28.627810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45216 (local address 127.0.98.60:39817)
0927 15:01:28.627925 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:28.627928 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:28.628360 (+ 432us) server_negotiation.cc:407] Connection header received
0927 15:01:28.628548 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:28.628552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:28.628599 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:28.628642 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:01:28.628947 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:28.629448 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:28.630887 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:28.631114 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:28.631173 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:28.631553 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:01:28.631631 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:01:28.631689 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:28.631789 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 15:01:28.631845 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:01:28.632085 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:28.627737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48496 (local address 127.0.98.61:43565)
0927 15:01:28.627862 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:28.627866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:28.628347 (+ 481us) server_negotiation.cc:407] Connection header received
0927 15:01:28.628436 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:28.628439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:28.628490 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:28.628552 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:01:28.628868 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:28.629611 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:28.630517 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:28.630752 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:28.630879 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:28.631607 (+ 728us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:01:28.631687 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:01:28.631728 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:28.631876 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 15:01:28.631937 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:01:28.632501 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:01:28.627832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46078 (local address 127.0.98.62:46587)
0927 15:01:28.628082 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 15:01:28.628086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:01:28.628100 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:01:28.628198 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:01:28.628201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:01:28.628380 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:01:28.628499 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:01:28.628709 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:28.629533 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:28.630880 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:01:28.631130 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:01:28.631207 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:01:28.631592 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:01:28.631962 (+ 370us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:01:28.632075 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 15:01:28.632308 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 15:01:28.632369 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":67,"threads_started":1}
I20260927 15:01:28.635484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 23012
I20260927 15:01:28.641510 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 23087
I20260927 15:01:28.646747 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDguobN/build/debug/bin/kudu with pid 23161
2026-09-27T15:01:28Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3085 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6195 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (284399 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST